2023-10-06T01:26:59.513 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-06T01:26:59.513 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --name yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi --verbose --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} --archive /home/teuthworker/archive/yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi/7412096 -- /home/teuthworker/archive/yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi/7412096/orig.config.yaml 2023-10-06T01:26:59.557 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-06T01:26:59.639 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi/7412096 branch: wip-yuri-testing-2023-10-05-0730-quincy description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} email: yweinste@redhat.com first_in_suite: false job_id: '7412096' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.4' overrides: admin_socket: branch: wip-yuri-testing-2023-10-05-0730-quincy ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 736ac5ba72cb47af7d6f6666b1e100f7e20a20e2 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 736ac5ba72cb47af7d6f6666b1e100f7e20a20e2 kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri-testing-2023-10-05-0730-quincy sha1: f1e315a9732d36c05ec316678e2397dc58561c7f owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 7519 sha1: 736ac5ba72cb47af7d6f6666b1e100f7e20a20e2 sleep_before_teardown: 0 subset: 111/12000 suite: fs suite_branch: wip-yuri-testing-2023-10-05-0730-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_f1e315a9732d36c05ec316678e2397dc58561c7f/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: f1e315a9732d36c05ec316678e2397dc58561c7f targets: smithi027.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKMVItaKZieNJGHWDSdhW5L1jqh1+xgxATt847HawcIgu7996RoKJqcfPy6oEBctp/7TfmFHn08ST1pb4RT6P/M= smithi124.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPlciKcQZmKpVk5mhkr/65Bra2WtrkfGj0rG/nL0nqbOlucdiZdD+gY5t7TaSup1xruJNSuNkeYmPg9GEdBjrmo= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: null - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_backtrace teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-10-05_23:52:50 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-06T01:26:59.639 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_f1e315a9732d36c05ec316678e2397dc58561c7f/qa; will attempt to use it 2023-10-06T01:26:59.640 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_f1e315a9732d36c05ec316678e2397dc58561c7f/qa/tasks 2023-10-06T01:26:59.640 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-06T01:26:59.642 INFO:teuthology.task.internal:Checking packages... 2023-10-06T01:26:59.665 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '736ac5ba72cb47af7d6f6666b1e100f7e20a20e2' 2023-10-06T01:26:59.665 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-06T01:26:59.665 INFO:teuthology.packaging:ref: None 2023-10-06T01:26:59.665 INFO:teuthology.packaging:tag: None 2023-10-06T01:26:59.665 INFO:teuthology.packaging:branch: wip-yuri-testing-2023-10-05-0730-quincy 2023-10-06T01:26:59.665 INFO:teuthology.packaging:sha1: 736ac5ba72cb47af7d6f6666b1e100f7e20a20e2 2023-10-06T01:26:59.666 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri-testing-2023-10-05-0730-quincy 2023-10-06T01:26:59.886 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-1296.g736ac5ba 2023-10-06T01:26:59.886 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-06T01:26:59.901 INFO:teuthology.task.internal:no buildpackages task found 2023-10-06T01:26:59.901 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-06T01:26:59.908 INFO:teuthology.task.internal:Saving configuration 2023-10-06T01:26:59.924 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-06T01:26:59.930 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-06T01:26:59.954 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi/7412096', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-06 01:20:35.180778', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKMVItaKZieNJGHWDSdhW5L1jqh1+xgxATt847HawcIgu7996RoKJqcfPy6oEBctp/7TfmFHn08ST1pb4RT6P/M='} 2023-10-06T01:26:59.978 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi124.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi/7412096', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-06 01:20:35.179337', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPlciKcQZmKpVk5mhkr/65Bra2WtrkfGj0rG/nL0nqbOlucdiZdD+gY5t7TaSup1xruJNSuNkeYmPg9GEdBjrmo='} 2023-10-06T01:26:59.978 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-06T01:26:59.984 INFO:teuthology.task.internal:roles: ubuntu@smithi027.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-10-06T01:26:59.984 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-10-06T01:26:59.984 INFO:teuthology.run_tasks:Running task console_log... 2023-10-06T01:27:00.045 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbfd48d9b80>, signals=[15]) 2023-10-06T01:27:00.045 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-06T01:27:00.052 INFO:teuthology.task.internal:Opening connections... 2023-10-06T01:27:00.052 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi027.front.sepia.ceph.com 2023-10-06T01:27:00.054 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-06T01:27:00.122 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi124.front.sepia.ceph.com 2023-10-06T01:27:00.123 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi124.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-06T01:27:00.188 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-06T01:27:00.196 DEBUG:teuthology.orchestra.run.smithi027:> uname -m 2023-10-06T01:27:00.214 INFO:teuthology.orchestra.run.smithi027.stdout:x86_64 2023-10-06T01:27:00.214 DEBUG:teuthology.orchestra.run.smithi027:> cat /etc/os-release 2023-10-06T01:27:00.271 INFO:teuthology.orchestra.run.smithi027.stdout:NAME="Red Hat Enterprise Linux" 2023-10-06T01:27:00.271 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION="8.4 (Ootpa)" 2023-10-06T01:27:00.271 INFO:teuthology.orchestra.run.smithi027.stdout:ID="rhel" 2023-10-06T01:27:00.271 INFO:teuthology.orchestra.run.smithi027.stdout:ID_LIKE="fedora" 2023-10-06T01:27:00.271 INFO:teuthology.orchestra.run.smithi027.stdout:VERSION_ID="8.4" 2023-10-06T01:27:00.271 INFO:teuthology.orchestra.run.smithi027.stdout:PLATFORM_ID="platform:el8" 2023-10-06T01:27:00.272 INFO:teuthology.orchestra.run.smithi027.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-10-06T01:27:00.272 INFO:teuthology.orchestra.run.smithi027.stdout:ANSI_COLOR="0;31" 2023-10-06T01:27:00.272 INFO:teuthology.orchestra.run.smithi027.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-10-06T01:27:00.272 INFO:teuthology.orchestra.run.smithi027.stdout:HOME_URL="https://www.redhat.com/" 2023-10-06T01:27:00.272 INFO:teuthology.orchestra.run.smithi027.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-06T01:27:00.272 INFO:teuthology.orchestra.run.smithi027.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-06T01:27:00.272 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:27:00.272 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-06T01:27:00.272 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-10-06T01:27:00.272 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-06T01:27:00.272 INFO:teuthology.orchestra.run.smithi027.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-10-06T01:27:00.273 INFO:teuthology.lock.ops:Updating smithi027.front.sepia.ceph.com on lock server 2023-10-06T01:27:00.296 DEBUG:teuthology.orchestra.run.smithi124:> uname -m 2023-10-06T01:27:00.312 INFO:teuthology.orchestra.run.smithi124.stdout:x86_64 2023-10-06T01:27:00.313 DEBUG:teuthology.orchestra.run.smithi124:> cat /etc/os-release 2023-10-06T01:27:00.368 INFO:teuthology.orchestra.run.smithi124.stdout:NAME="Red Hat Enterprise Linux" 2023-10-06T01:27:00.368 INFO:teuthology.orchestra.run.smithi124.stdout:VERSION="8.4 (Ootpa)" 2023-10-06T01:27:00.368 INFO:teuthology.orchestra.run.smithi124.stdout:ID="rhel" 2023-10-06T01:27:00.369 INFO:teuthology.orchestra.run.smithi124.stdout:ID_LIKE="fedora" 2023-10-06T01:27:00.369 INFO:teuthology.orchestra.run.smithi124.stdout:VERSION_ID="8.4" 2023-10-06T01:27:00.369 INFO:teuthology.orchestra.run.smithi124.stdout:PLATFORM_ID="platform:el8" 2023-10-06T01:27:00.369 INFO:teuthology.orchestra.run.smithi124.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-10-06T01:27:00.369 INFO:teuthology.orchestra.run.smithi124.stdout:ANSI_COLOR="0;31" 2023-10-06T01:27:00.369 INFO:teuthology.orchestra.run.smithi124.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-10-06T01:27:00.369 INFO:teuthology.orchestra.run.smithi124.stdout:HOME_URL="https://www.redhat.com/" 2023-10-06T01:27:00.369 INFO:teuthology.orchestra.run.smithi124.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-10-06T01:27:00.369 INFO:teuthology.orchestra.run.smithi124.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-06T01:27:00.369 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-10-06T01:27:00.370 INFO:teuthology.orchestra.run.smithi124.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-06T01:27:00.370 INFO:teuthology.orchestra.run.smithi124.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-10-06T01:27:00.370 INFO:teuthology.orchestra.run.smithi124.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-10-06T01:27:00.370 INFO:teuthology.orchestra.run.smithi124.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-10-06T01:27:00.370 INFO:teuthology.lock.ops:Updating smithi124.front.sepia.ceph.com on lock server 2023-10-06T01:27:00.388 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-06T01:27:00.397 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-06T01:27:00.402 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-06T01:27:00.402 DEBUG:teuthology.orchestra.run.smithi027:> test '!' -e /home/ubuntu/cephtest 2023-10-06T01:27:00.404 DEBUG:teuthology.orchestra.run.smithi124:> test '!' -e /home/ubuntu/cephtest 2023-10-06T01:27:00.422 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-06T01:27:00.428 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-06T01:27:00.428 DEBUG:teuthology.orchestra.run.smithi027:> test -z $(ls -A /var/lib/ceph) 2023-10-06T01:27:00.462 DEBUG:teuthology.orchestra.run.smithi124:> test -z $(ls -A /var/lib/ceph) 2023-10-06T01:27:00.476 INFO:teuthology.orchestra.run.smithi027.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-10-06T01:27:00.478 INFO:teuthology.orchestra.run.smithi124.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-10-06T01:27:00.478 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-06T01:27:00.528 INFO:teuthology.run_tasks:Running task kernel... 2023-10-06T01:27:00.542 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2023-10-06T01:27:00.542 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.2': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}, 'client.3': {'sha1': 'distro'}} 2023-10-06T01:27:00.543 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2023-10-06T01:27:00.543 DEBUG:teuthology.orchestra.run.smithi027:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-06T01:27:00.544 DEBUG:teuthology.orchestra.run.smithi124:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-06T01:27:00.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T01:27:00.558 DEBUG:teuthology.orchestra.run.smithi124:> uname -r 2023-10-06T01:27:00.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T01:27:00.560 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2023-10-06T01:27:00.613 INFO:teuthology.orchestra.run.smithi124.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-10-06T01:27:00.613 INFO:teuthology.task.kernel:Running kernel on smithi124: 4.18.0-305.25.1.el8_4.x86_64 2023-10-06T01:27:00.613 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum install -y kernel 2023-10-06T01:27:00.617 INFO:teuthology.orchestra.run.smithi027.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-10-06T01:27:00.617 INFO:teuthology.task.kernel:Running kernel on smithi027: 4.18.0-305.25.1.el8_4.x86_64 2023-10-06T01:27:00.617 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum install -y kernel 2023-10-06T01:27:02.037 INFO:teuthology.orchestra.run.smithi027.stdout:Updating Subscription Management repositories. 2023-10-06T01:27:02.038 INFO:teuthology.orchestra.run.smithi027.stdout:Unable to read consumer identity 2023-10-06T01:27:02.050 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2023-10-06T01:27:02.050 INFO:teuthology.orchestra.run.smithi124.stdout:Unable to read consumer identity 2023-10-06T01:27:03.457 INFO:teuthology.orchestra.run.smithi027.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2023-10-06T01:27:03.462 INFO:teuthology.orchestra.run.smithi124.stdout:Copr repo for python3-asyncssh owned by ceph 16 kB/s | 3.5 kB 00:00 2023-10-06T01:27:04.050 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 29 MB/s | 16 MB 00:00 2023-10-06T01:27:04.196 INFO:teuthology.orchestra.run.smithi124.stdout:Extra Packages for Enterprise Linux 23 MB/s | 16 MB 00:00 2023-10-06T01:27:07.475 INFO:teuthology.orchestra.run.smithi124.stdout:lab-extras 1.7 MB/s | 24 kB 00:00 2023-10-06T01:27:07.541 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 4.7 MB/s | 24 kB 00:00 2023-10-06T01:27:08.161 INFO:teuthology.orchestra.run.smithi124.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-10-06T01:27:08.161 INFO:teuthology.orchestra.run.smithi124.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-10-06T01:27:08.162 INFO:teuthology.orchestra.run.smithi124.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-10-06T01:27:08.334 INFO:teuthology.orchestra.run.smithi124.stdout:Dependencies resolved. 2023-10-06T01:27:08.335 INFO:teuthology.orchestra.run.smithi124.stdout:Nothing to do. 2023-10-06T01:27:08.335 INFO:teuthology.orchestra.run.smithi124.stdout:Complete! 2023-10-06T01:27:08.348 INFO:teuthology.orchestra.run.smithi027.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-10-06T01:27:08.348 INFO:teuthology.orchestra.run.smithi027.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-10-06T01:27:08.348 INFO:teuthology.orchestra.run.smithi027.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-10-06T01:27:08.426 DEBUG:teuthology.orchestra.run.smithi124:> echo no | sudo yum reinstall kernel || true 2023-10-06T01:27:08.490 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-10-06T01:27:08.491 INFO:teuthology.orchestra.run.smithi027.stdout:Nothing to do. 2023-10-06T01:27:08.492 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-10-06T01:27:08.622 DEBUG:teuthology.orchestra.run.smithi027:> echo no | sudo yum reinstall kernel || true 2023-10-06T01:27:08.768 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2023-10-06T01:27:08.768 INFO:teuthology.orchestra.run.smithi124.stdout:Unable to read consumer identity 2023-10-06T01:27:08.987 INFO:teuthology.orchestra.run.smithi027.stdout:Updating Subscription Management repositories. 2023-10-06T01:27:08.987 INFO:teuthology.orchestra.run.smithi027.stdout:Unable to read consumer identity 2023-10-06T01:27:09.063 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:00:02 ago on Fri 06 Oct 2023 01:27:07 AM UTC. 2023-10-06T01:27:09.203 INFO:teuthology.orchestra.run.smithi124.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-06T01:27:09.204 INFO:teuthology.orchestra.run.smithi124.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-06T01:27:09.204 INFO:teuthology.orchestra.run.smithi124.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-06T01:27:09.228 INFO:teuthology.orchestra.run.smithi124.stderr:Error: No packages marked for reinstall. 2023-10-06T01:27:09.277 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum reinstall -y kernel || true 2023-10-06T01:27:09.292 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:02 ago on Fri 06 Oct 2023 01:27:07 AM UTC. 2023-10-06T01:27:09.432 INFO:teuthology.orchestra.run.smithi027.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-06T01:27:09.432 INFO:teuthology.orchestra.run.smithi027.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-06T01:27:09.432 INFO:teuthology.orchestra.run.smithi027.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-06T01:27:09.460 INFO:teuthology.orchestra.run.smithi027.stderr:Error: No packages marked for reinstall. 2023-10-06T01:27:09.512 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum reinstall -y kernel || true 2023-10-06T01:27:09.616 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2023-10-06T01:27:09.617 INFO:teuthology.orchestra.run.smithi124.stdout:Unable to read consumer identity 2023-10-06T01:27:09.874 INFO:teuthology.orchestra.run.smithi027.stdout:Updating Subscription Management repositories. 2023-10-06T01:27:09.874 INFO:teuthology.orchestra.run.smithi027.stdout:Unable to read consumer identity 2023-10-06T01:27:09.899 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:00:02 ago on Fri 06 Oct 2023 01:27:07 AM UTC. 2023-10-06T01:27:10.036 INFO:teuthology.orchestra.run.smithi124.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-06T01:27:10.036 INFO:teuthology.orchestra.run.smithi124.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-06T01:27:10.036 INFO:teuthology.orchestra.run.smithi124.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-06T01:27:10.052 INFO:teuthology.orchestra.run.smithi124.stderr:Error: No packages marked for reinstall. 2023-10-06T01:27:10.101 DEBUG:teuthology.orchestra.run.smithi124:> rpm -q kernel | sort -rV | head -n 1 2023-10-06T01:27:10.176 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:00:03 ago on Fri 06 Oct 2023 01:27:07 AM UTC. 2023-10-06T01:27:10.225 INFO:teuthology.orchestra.run.smithi124.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-06T01:27:10.226 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-06T01:27:10.226 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-10-06T01:27:10.226 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-06T01:27:10.226 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-06T01:27:10.226 INFO:teuthology.task.kernel:Installing distro kernel on client.1... 2023-10-06T01:27:10.226 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi124.front.sepia.ceph.com, path=None, version=distro) 2023-10-06T01:27:10.227 DEBUG:teuthology.orchestra.run.smithi124:> rpm -q kernel | sort -rV | head -n 1 2023-10-06T01:27:10.331 INFO:teuthology.orchestra.run.smithi027.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-10-06T01:27:10.331 INFO:teuthology.orchestra.run.smithi027.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-06T01:27:10.331 INFO:teuthology.orchestra.run.smithi027.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-10-06T01:27:10.351 INFO:teuthology.orchestra.run.smithi124.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-06T01:27:10.351 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-06T01:27:10.351 DEBUG:teuthology.orchestra.run.smithi124:> sudo rpm -qi grub2-tools 2023-10-06T01:27:10.367 INFO:teuthology.orchestra.run.smithi027.stderr:Error: No packages marked for reinstall. 2023-10-06T01:27:10.378 INFO:teuthology.orchestra.run.smithi124.stdout:Name : grub2-tools 2023-10-06T01:27:10.378 INFO:teuthology.orchestra.run.smithi124.stdout:Epoch : 1 2023-10-06T01:27:10.378 INFO:teuthology.orchestra.run.smithi124.stdout:Version : 2.02 2023-10-06T01:27:10.378 INFO:teuthology.orchestra.run.smithi124.stdout:Release : 99.el8_4.1 2023-10-06T01:27:10.378 INFO:teuthology.orchestra.run.smithi124.stdout:Architecture: x86_64 2023-10-06T01:27:10.378 INFO:teuthology.orchestra.run.smithi124.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-10-06T01:27:10.378 INFO:teuthology.orchestra.run.smithi124.stdout:Group : System Environment/Base 2023-10-06T01:27:10.378 INFO:teuthology.orchestra.run.smithi124.stdout:Size : 9264631 2023-10-06T01:27:10.378 INFO:teuthology.orchestra.run.smithi124.stdout:License : GPLv3+ 2023-10-06T01:27:10.378 INFO:teuthology.orchestra.run.smithi124.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-10-06T01:27:10.378 INFO:teuthology.orchestra.run.smithi124.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-10-06T01:27:10.379 INFO:teuthology.orchestra.run.smithi124.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-10-06T01:27:10.379 INFO:teuthology.orchestra.run.smithi124.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-10-06T01:27:10.379 INFO:teuthology.orchestra.run.smithi124.stdout:Relocations : (not relocatable) 2023-10-06T01:27:10.379 INFO:teuthology.orchestra.run.smithi124.stdout:Packager : Red Hat, Inc. 2023-10-06T01:27:10.379 INFO:teuthology.orchestra.run.smithi124.stdout:Vendor : Red Hat, Inc. 2023-10-06T01:27:10.379 INFO:teuthology.orchestra.run.smithi124.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-06T01:27:10.379 INFO:teuthology.orchestra.run.smithi124.stdout:Summary : Support tools for GRUB. 2023-10-06T01:27:10.379 INFO:teuthology.orchestra.run.smithi124.stdout:Description : 2023-10-06T01:27:10.379 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-10-06T01:27:10.379 INFO:teuthology.orchestra.run.smithi124.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-06T01:27:10.379 INFO:teuthology.orchestra.run.smithi124.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-06T01:27:10.379 INFO:teuthology.orchestra.run.smithi124.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-06T01:27:10.380 INFO:teuthology.orchestra.run.smithi124.stdout:hardware devices. 2023-10-06T01:27:10.380 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-10-06T01:27:10.380 INFO:teuthology.orchestra.run.smithi124.stdout:This subpackage provides tools for support of all platforms. 2023-10-06T01:27:10.380 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-06T01:27:10.381 INFO:teuthology.task.kernel:Updating grub on smithi124 to boot 4.18.0-425.3.1.el8.x86_64 2023-10-06T01:27:10.381 DEBUG:teuthology.orchestra.run.smithi124:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-06T01:27:10.424 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q kernel | sort -rV | head -n 1 2023-10-06T01:27:10.554 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-06T01:27:10.554 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-06T01:27:10.554 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-10-06T01:27:10.554 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-06T01:27:10.555 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-06T01:27:10.555 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2023-10-06T01:27:10.555 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi027.front.sepia.ceph.com, path=None, version=distro) 2023-10-06T01:27:10.555 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q kernel | sort -rV | head -n 1 2023-10-06T01:27:10.686 INFO:teuthology.orchestra.run.smithi027.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-10-06T01:27:10.686 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-10-06T01:27:10.686 DEBUG:teuthology.orchestra.run.smithi027:> sudo rpm -qi grub2-tools 2023-10-06T01:27:10.714 INFO:teuthology.orchestra.run.smithi027.stdout:Name : grub2-tools 2023-10-06T01:27:10.715 INFO:teuthology.orchestra.run.smithi027.stdout:Epoch : 1 2023-10-06T01:27:10.715 INFO:teuthology.orchestra.run.smithi027.stdout:Version : 2.02 2023-10-06T01:27:10.715 INFO:teuthology.orchestra.run.smithi027.stdout:Release : 99.el8_4.1 2023-10-06T01:27:10.715 INFO:teuthology.orchestra.run.smithi027.stdout:Architecture: x86_64 2023-10-06T01:27:10.715 INFO:teuthology.orchestra.run.smithi027.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-10-06T01:27:10.715 INFO:teuthology.orchestra.run.smithi027.stdout:Group : System Environment/Base 2023-10-06T01:27:10.715 INFO:teuthology.orchestra.run.smithi027.stdout:Size : 9264631 2023-10-06T01:27:10.715 INFO:teuthology.orchestra.run.smithi027.stdout:License : GPLv3+ 2023-10-06T01:27:10.715 INFO:teuthology.orchestra.run.smithi027.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-10-06T01:27:10.715 INFO:teuthology.orchestra.run.smithi027.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-10-06T01:27:10.715 INFO:teuthology.orchestra.run.smithi027.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-10-06T01:27:10.716 INFO:teuthology.orchestra.run.smithi027.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-10-06T01:27:10.716 INFO:teuthology.orchestra.run.smithi027.stdout:Relocations : (not relocatable) 2023-10-06T01:27:10.716 INFO:teuthology.orchestra.run.smithi027.stdout:Packager : Red Hat, Inc. 2023-10-06T01:27:10.716 INFO:teuthology.orchestra.run.smithi027.stdout:Vendor : Red Hat, Inc. 2023-10-06T01:27:10.716 INFO:teuthology.orchestra.run.smithi027.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-06T01:27:10.716 INFO:teuthology.orchestra.run.smithi027.stdout:Summary : Support tools for GRUB. 2023-10-06T01:27:10.716 INFO:teuthology.orchestra.run.smithi027.stdout:Description : 2023-10-06T01:27:10.716 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:27:10.716 INFO:teuthology.orchestra.run.smithi027.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-06T01:27:10.716 INFO:teuthology.orchestra.run.smithi027.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-06T01:27:10.716 INFO:teuthology.orchestra.run.smithi027.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-06T01:27:10.716 INFO:teuthology.orchestra.run.smithi027.stdout:hardware devices. 2023-10-06T01:27:10.717 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:27:10.717 INFO:teuthology.orchestra.run.smithi027.stdout:This subpackage provides tools for support of all platforms. 2023-10-06T01:27:10.717 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-06T01:27:10.718 INFO:teuthology.task.kernel:Updating grub on smithi027 to boot 4.18.0-425.3.1.el8.x86_64 2023-10-06T01:27:10.718 DEBUG:teuthology.orchestra.run.smithi027:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-06T01:27:11.571 INFO:teuthology.orchestra.run.smithi124.stderr:Generating grub configuration file ... 2023-10-06T01:27:12.076 INFO:teuthology.orchestra.run.smithi027.stderr:Generating grub configuration file ... 2023-10-06T01:27:13.338 INFO:teuthology.orchestra.run.smithi124.stderr:done 2023-10-06T01:27:13.340 DEBUG:teuthology.orchestra.run.smithi124:> mktemp 2023-10-06T01:27:13.354 INFO:teuthology.orchestra.run.smithi124.stdout:/tmp/tmp.dkndkTht72 2023-10-06T01:27:13.355 DEBUG:teuthology.orchestra.run.smithi124:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.dkndkTht72 2023-10-06T01:27:13.424 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod 0666 /tmp/tmp.dkndkTht72 2023-10-06T01:27:13.588 DEBUG:teuthology.orchestra.remote:smithi124:/tmp/tmp.dkndkTht72 is 6KB 2023-10-06T01:27:13.636 DEBUG:teuthology.orchestra.run.smithi124:> rm -fr /tmp/tmp.dkndkTht72 2023-10-06T01:27:13.650 DEBUG:teuthology.orchestra.run.smithi124:> sudo /bin/ls /boot/loader/entries || true 2023-10-06T01:27:13.734 INFO:teuthology.orchestra.run.smithi124.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-10-06T01:27:13.734 INFO:teuthology.orchestra.run.smithi124.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-10-06T01:27:13.734 INFO:teuthology.orchestra.run.smithi124.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-10-06T01:27:13.734 INFO:teuthology.orchestra.run.smithi124.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-10-06T01:27:13.736 DEBUG:teuthology.orchestra.run.smithi124:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-10-06T01:27:13.849 DEBUG:teuthology.orchestra.run.smithi124:> sudo shutdown -r now 2023-10-06T01:27:14.243 INFO:teuthology.orchestra.run.smithi027.stderr:done 2023-10-06T01:27:14.245 DEBUG:teuthology.orchestra.run.smithi027:> mktemp 2023-10-06T01:27:14.259 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/tmp.5tExCLgTSG 2023-10-06T01:27:14.260 DEBUG:teuthology.orchestra.run.smithi027:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.5tExCLgTSG 2023-10-06T01:27:14.335 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0666 /tmp/tmp.5tExCLgTSG 2023-10-06T01:27:14.587 DEBUG:teuthology.orchestra.remote:smithi027:/tmp/tmp.5tExCLgTSG is 6KB 2023-10-06T01:27:14.635 DEBUG:teuthology.orchestra.run.smithi027:> rm -fr /tmp/tmp.5tExCLgTSG 2023-10-06T01:27:14.649 DEBUG:teuthology.orchestra.run.smithi027:> sudo /bin/ls /boot/loader/entries || true 2023-10-06T01:27:14.711 INFO:teuthology.orchestra.run.smithi027.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-10-06T01:27:14.711 INFO:teuthology.orchestra.run.smithi027.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-10-06T01:27:14.712 INFO:teuthology.orchestra.run.smithi027.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-10-06T01:27:14.712 INFO:teuthology.orchestra.run.smithi027.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-10-06T01:27:14.713 DEBUG:teuthology.orchestra.run.smithi027:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-10-06T01:27:14.886 DEBUG:teuthology.orchestra.run.smithi027:> sudo shutdown -r now 2023-10-06T01:27:43.879 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2023-10-06T01:27:43.880 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-10-06T01:27:43.881 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi124.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-06T01:27:44.889 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-10-06T01:27:44.890 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-10-06T01:27:44.890 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-06T01:28:02.430 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.124 2023-10-06T01:28:03.450 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2023-10-06T01:28:11.439 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-10-06T01:28:11.440 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi124.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-06T01:28:12.451 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-10-06T01:28:12.451 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-06T01:28:14.494 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.124 2023-10-06T01:28:15.514 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2023-10-06T01:28:26.506 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-10-06T01:28:26.507 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi124.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-06T01:28:27.516 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-10-06T01:28:27.517 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-06T01:28:30.586 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2023-10-06T01:28:44.922 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.124 2023-10-06T01:28:45.588 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-10-06T01:28:45.589 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-06T01:28:48.666 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2023-10-06T01:28:59.930 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-10-06T01:28:59.931 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi124.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-06T01:29:00.266 DEBUG:teuthology.orchestra.run.smithi124:> true 2023-10-06T01:29:00.552 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-10-06T01:29:00.552 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "4.18.0-425.3.1.el8.x86_64"... 2023-10-06T01:29:00.553 DEBUG:teuthology.orchestra.run.smithi124:> uname -r 2023-10-06T01:29:00.567 INFO:teuthology.orchestra.run.smithi124.stdout:4.18.0-425.3.1.el8.x86_64 2023-10-06T01:29:00.567 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-10-06T01:29:00.567 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-06T01:29:00.567 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-10-06T01:29:01.569 DEBUG:teuthology.parallel:result is None 2023-10-06T01:29:06.668 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-10-06T01:29:06.668 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-06T01:29:25.114 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.27 2023-10-06T01:29:46.115 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-10-06T01:29:46.116 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-06T01:29:46.415 DEBUG:teuthology.orchestra.run.smithi027:> true 2023-10-06T01:29:46.636 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-10-06T01:29:46.636 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "4.18.0-425.3.1.el8.x86_64"... 2023-10-06T01:29:46.636 DEBUG:teuthology.orchestra.run.smithi027:> uname -r 2023-10-06T01:29:46.692 INFO:teuthology.orchestra.run.smithi027.stdout:4.18.0-425.3.1.el8.x86_64 2023-10-06T01:29:46.692 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-10-06T01:29:46.692 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-06T01:29:46.692 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-10-06T01:29:47.693 DEBUG:teuthology.parallel:result is None 2023-10-06T01:29:47.694 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-06T01:29:47.701 INFO:teuthology.task.internal:Creating test directory... 2023-10-06T01:29:47.701 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-06T01:29:47.704 DEBUG:teuthology.orchestra.run.smithi124:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-06T01:29:47.722 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-06T01:29:47.728 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-06T01:29:47.735 INFO:teuthology.task.internal:Creating archive directory... 2023-10-06T01:29:47.735 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-06T01:29:47.760 DEBUG:teuthology.orchestra.run.smithi124:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-06T01:29:47.791 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-06T01:29:47.797 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-06T01:29:47.797 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-06T01:29:47.822 DEBUG:teuthology.orchestra.run.smithi124:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-06T01:29:47.910 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-06T01:29:47.929 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-06T01:29:47.945 INFO:teuthology.orchestra.run.smithi124.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-06T01:29:47.964 INFO:teuthology.orchestra.run.smithi124.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-06T01:29:47.966 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-06T01:29:47.973 INFO:teuthology.task.internal:Configuring sudo... 2023-10-06T01:29:47.973 DEBUG:teuthology.orchestra.run.smithi027:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-06T01:29:47.977 DEBUG:teuthology.orchestra.run.smithi124:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-06T01:29:48.003 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-06T01:29:48.012 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-06T01:29:48.012 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-06T01:29:48.042 DEBUG:teuthology.orchestra.run.smithi124:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-06T01:29:48.058 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-06T01:29:48.137 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-06T01:29:48.193 DEBUG:teuthology.orchestra.run.smithi027:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-06T01:29:48.272 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-06T01:29:48.315 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-06T01:29:48.315 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-06T01:29:48.380 DEBUG:teuthology.orchestra.run.smithi124:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-06T01:29:48.421 DEBUG:teuthology.orchestra.run.smithi124:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-06T01:29:48.480 DEBUG:teuthology.orchestra.run.smithi124:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-06T01:29:48.563 DEBUG:teuthology.orchestra.run.smithi124:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-06T01:29:48.603 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-10-06T01:29:48.603 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-06T01:29:48.667 DEBUG:teuthology.orchestra.run.smithi027:> sudo service rsyslog restart 2023-10-06T01:29:48.669 DEBUG:teuthology.orchestra.run.smithi124:> sudo service rsyslog restart 2023-10-06T01:29:48.717 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-06T01:29:48.787 INFO:teuthology.orchestra.run.smithi124.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-06T01:29:49.139 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-06T01:29:49.147 INFO:teuthology.task.internal:Starting timer... 2023-10-06T01:29:49.147 INFO:teuthology.run_tasks:Running task pcp... 2023-10-06T01:29:49.157 INFO:teuthology.run_tasks:Running task selinux... 2023-10-06T01:29:49.167 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']} 2023-10-06T01:29:49.168 DEBUG:teuthology.orchestra.run.smithi027:> sudo service auditd rotate 2023-10-06T01:29:49.235 INFO:teuthology.orchestra.run.smithi027.stdout:Rotating logs: [ OK ] 2023-10-06T01:29:49.237 DEBUG:teuthology.orchestra.run.smithi124:> sudo service auditd rotate 2023-10-06T01:29:49.304 INFO:teuthology.orchestra.run.smithi124.stdout:Rotating logs: [ OK ] 2023-10-06T01:29:49.306 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-06T01:29:49.306 DEBUG:teuthology.orchestra.run.smithi027:> /usr/sbin/getenforce 2023-10-06T01:29:49.326 INFO:teuthology.orchestra.run.smithi027.stdout:Permissive 2023-10-06T01:29:49.326 DEBUG:teuthology.orchestra.run.smithi124:> /usr/sbin/getenforce 2023-10-06T01:29:49.345 INFO:teuthology.orchestra.run.smithi124.stdout:Permissive 2023-10-06T01:29:49.345 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi027.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi124.front.sepia.ceph.com': 'permissive'} 2023-10-06T01:29:49.345 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-06T01:29:49.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T01:29:49.394 DEBUG:teuthology.orchestra.run.smithi124:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-06T01:29:49.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T01:29:49.419 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-06T01:29:49.420 DEBUG:teuthology.orchestra.run.smithi027:> sudo /usr/sbin/setenforce permissive 2023-10-06T01:29:49.461 DEBUG:teuthology.orchestra.run.smithi124:> sudo /usr/sbin/setenforce permissive 2023-10-06T01:29:49.486 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-06T01:29:49.495 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-06T01:29:49.500 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-06T01:29:49.607 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-06T01:29:49.624 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-06T01:29:49.625 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi027.front.sepia.ceph.com,smithi124.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-06T01:38:11.169 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi027.front.sepia.ceph.com'), Remote(name='ubuntu@smithi124.front.sepia.ceph.com')] 2023-10-06T01:38:11.170 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-10-06T01:38:11.171 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi027.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-06T01:38:11.243 DEBUG:teuthology.orchestra.run.smithi027:> true 2023-10-06T01:38:11.310 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi027.front.sepia.ceph.com' 2023-10-06T01:38:11.310 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-10-06T01:38:11.310 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi124.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-06T01:38:11.378 DEBUG:teuthology.orchestra.run.smithi124:> true 2023-10-06T01:38:11.401 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi124.front.sepia.ceph.com' 2023-10-06T01:38:11.401 INFO:teuthology.run_tasks:Running task clock... 2023-10-06T01:38:11.413 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-06T01:38:11.413 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-06T01:38:11.413 DEBUG:teuthology.orchestra.run.smithi027:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-06T01:38:11.415 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-06T01:38:11.415 DEBUG:teuthology.orchestra.run.smithi124:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-06T01:38:11.444 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-06T01:38:11.459 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-06T01:38:11.470 INFO:teuthology.orchestra.run.smithi124.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-06T01:38:11.484 INFO:teuthology.orchestra.run.smithi124.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-06T01:38:11.501 INFO:teuthology.orchestra.run.smithi027.stderr:sudo: ntpd: command not found 2023-10-06T01:38:11.512 INFO:teuthology.orchestra.run.smithi027.stdout:506 Cannot talk to daemon 2023-10-06T01:38:11.515 INFO:teuthology.orchestra.run.smithi124.stderr:sudo: ntpd: command not found 2023-10-06T01:38:11.526 INFO:teuthology.orchestra.run.smithi124.stdout:506 Cannot talk to daemon 2023-10-06T01:38:11.527 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-06T01:38:11.539 INFO:teuthology.orchestra.run.smithi124.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-06T01:38:11.541 INFO:teuthology.orchestra.run.smithi027.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-06T01:38:11.553 INFO:teuthology.orchestra.run.smithi124.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-06T01:38:11.613 INFO:teuthology.orchestra.run.smithi027.stderr:bash: ntpq: command not found 2023-10-06T01:38:11.616 INFO:teuthology.orchestra.run.smithi027.stdout:210 Number of sources = 4 2023-10-06T01:38:11.616 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-06T01:38:11.616 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2023-10-06T01:38:11.616 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-06T01:38:11.616 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-06T01:38:11.617 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-06T01:38:11.617 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-06T01:38:11.617 INFO:teuthology.orchestra.run.smithi124.stderr:bash: ntpq: command not found 2023-10-06T01:38:11.618 INFO:teuthology.orchestra.run.smithi124.stdout:210 Number of sources = 4 2023-10-06T01:38:11.619 INFO:teuthology.orchestra.run.smithi124.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-06T01:38:11.619 INFO:teuthology.orchestra.run.smithi124.stdout:=============================================================================== 2023-10-06T01:38:11.619 INFO:teuthology.orchestra.run.smithi124.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-06T01:38:11.619 INFO:teuthology.orchestra.run.smithi124.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-06T01:38:11.619 INFO:teuthology.orchestra.run.smithi124.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-06T01:38:11.619 INFO:teuthology.orchestra.run.smithi124.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-06T01:38:11.619 INFO:teuthology.run_tasks:Running task install... 2023-10-06T01:38:11.628 DEBUG:teuthology.task.install:project ceph 2023-10-06T01:38:11.628 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '736ac5ba72cb47af7d6f6666b1e100f7e20a20e2'}} 2023-10-06T01:38:11.628 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '736ac5ba72cb47af7d6f6666b1e100f7e20a20e2'} 2023-10-06T01:38:11.628 INFO:teuthology.task.install:Using flavor: default 2023-10-06T01:38:11.634 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-06T01:38:11.634 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-10-06T01:38:11.634 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '736ac5ba72cb47af7d6f6666b1e100f7e20a20e2', 'tag': None, 'wait_for_package': False} 2023-10-06T01:38:11.635 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=736ac5ba72cb47af7d6f6666b1e100f7e20a20e2 2023-10-06T01:38:11.636 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '736ac5ba72cb47af7d6f6666b1e100f7e20a20e2', 'tag': None, 'wait_for_package': False} 2023-10-06T01:38:11.637 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=736ac5ba72cb47af7d6f6666b1e100f7e20a20e2 2023-10-06T01:38:11.777 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/centos/8/flavors/default/ 2023-10-06T01:38:11.777 INFO:teuthology.task.install.rpm:Package version is 17.2.6-1296.g736ac5ba 2023-10-06T01:38:11.868 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/centos/8/flavors/default/ 2023-10-06T01:38:11.869 INFO:teuthology.task.install.rpm:Package version is 17.2.6-1296.g736ac5ba 2023-10-06T01:38:11.956 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-06T01:38:11.956 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-06T01:38:11.957 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-06T01:38:11.974 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-yuri-testing-2023-10-05-0730-quincy/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-06T01:38:11.975 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-10-06T01:38:11.975 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-06T01:38:11.983 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2023-10-06T01:38:11.983 DEBUG:teuthology.orchestra.run.smithi027:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-06T01:38:12.001 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2023-10-06T01:38:12.001 DEBUG:teuthology.orchestra.run.smithi124:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/736ac5ba72cb47af7d6f6666b1e100f7e20a20e2/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-06T01:38:12.049 DEBUG:teuthology.orchestra.run.smithi027:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-06T01:38:12.066 DEBUG:teuthology.orchestra.run.smithi124:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-06T01:38:12.124 DEBUG:teuthology.orchestra.run.smithi027:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-06T01:38:12.140 DEBUG:teuthology.orchestra.run.smithi124:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-06T01:38:12.188 INFO:teuthology.orchestra.run.smithi027.stdout:check_obsoletes = 1 2023-10-06T01:38:12.190 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum clean all 2023-10-06T01:38:12.203 INFO:teuthology.orchestra.run.smithi124.stdout:check_obsoletes = 1 2023-10-06T01:38:12.204 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum clean all 2023-10-06T01:38:12.593 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2023-10-06T01:38:12.602 INFO:teuthology.orchestra.run.smithi027.stdout:Updating Subscription Management repositories. 2023-10-06T01:38:12.916 INFO:teuthology.orchestra.run.smithi124.stdout:44 files removed 2023-10-06T01:38:12.926 INFO:teuthology.orchestra.run.smithi027.stdout:44 files removed 2023-10-06T01:38:12.968 DEBUG:teuthology.orchestra.run.smithi124:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2023-10-06T01:38:12.977 DEBUG:teuthology.orchestra.run.smithi027:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2023-10-06T01:38:13.315 INFO:teuthology.orchestra.run.smithi124.stdout:Updating Subscription Management repositories. 2023-10-06T01:38:13.346 INFO:teuthology.orchestra.run.smithi027.stdout:Updating Subscription Management repositories. 2023-10-06T01:38:14.212 INFO:teuthology.orchestra.run.smithi124.stdout:ceph packages for x86_64 182 kB/s | 78 kB 00:00 2023-10-06T01:38:14.243 INFO:teuthology.orchestra.run.smithi027.stdout:ceph packages for x86_64 181 kB/s | 78 kB 00:00 2023-10-06T01:38:14.557 INFO:teuthology.orchestra.run.smithi027.stdout:ceph noarch packages 34 kB/s | 9.2 kB 00:00 2023-10-06T01:38:14.612 INFO:teuthology.orchestra.run.smithi124.stdout:ceph noarch packages 26 kB/s | 9.2 kB 00:00 2023-10-06T01:38:14.862 INFO:teuthology.orchestra.run.smithi027.stdout:ceph source packages 5.3 kB/s | 1.4 kB 00:00 2023-10-06T01:38:14.994 INFO:teuthology.orchestra.run.smithi124.stdout:ceph source packages 4.1 kB/s | 1.4 kB 00:00 2023-10-06T01:38:15.017 INFO:teuthology.orchestra.run.smithi027.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2023-10-06T01:38:15.151 INFO:teuthology.orchestra.run.smithi124.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2023-10-06T01:38:17.263 INFO:teuthology.orchestra.run.smithi124.stdout:Extra Packages for Enterprise Linux 7.6 MB/s | 16 MB 00:02 2023-10-06T01:38:18.117 INFO:teuthology.orchestra.run.smithi027.stdout:Extra Packages for Enterprise Linux 5.2 MB/s | 16 MB 00:03 2023-10-06T01:38:20.608 INFO:teuthology.orchestra.run.smithi124.stdout:lab-extras 4.3 MB/s | 24 kB 00:00 2023-10-06T01:38:20.994 INFO:teuthology.orchestra.run.smithi124.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 126 MB/s | 38 MB 00:00 2023-10-06T01:38:21.568 INFO:teuthology.orchestra.run.smithi027.stdout:lab-extras 4.4 MB/s | 24 kB 00:00 2023-10-06T01:38:22.009 INFO:teuthology.orchestra.run.smithi027.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 108 MB/s | 38 MB 00:00 2023-10-06T01:38:26.441 INFO:teuthology.orchestra.run.smithi124.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 120 MB/s | 33 MB 00:00 2023-10-06T01:38:27.969 INFO:teuthology.orchestra.run.smithi027.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 104 MB/s | 33 MB 00:00 2023-10-06T01:38:34.448 INFO:teuthology.orchestra.run.smithi124.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-06T01:38:34.448 INFO:teuthology.orchestra.run.smithi124.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-06T01:38:34.459 INFO:teuthology.orchestra.run.smithi124.stdout:Package NetworkManager-1:1.30.0-13.el8_4.x86_64 is already installed. 2023-10-06T01:38:34.461 INFO:teuthology.orchestra.run.smithi124.stdout:Package iproute-5.9.0-4.el8.x86_64 is already installed. 2023-10-06T01:38:34.461 INFO:teuthology.orchestra.run.smithi124.stdout:Package util-linux-2.32.1-27.el8.x86_64 is already installed. 2023-10-06T01:38:34.461 INFO:teuthology.orchestra.run.smithi124.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-10-06T01:38:34.463 INFO:teuthology.orchestra.run.smithi124.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-10-06T01:38:34.465 INFO:teuthology.orchestra.run.smithi124.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2023-10-06T01:38:34.466 INFO:teuthology.orchestra.run.smithi124.stdout:Package xfsprogs-5.0.0-8.el8.x86_64 is already installed. 2023-10-06T01:38:34.962 INFO:teuthology.orchestra.run.smithi124.stdout:Dependencies resolved. 2023-10-06T01:38:34.973 INFO:teuthology.orchestra.run.smithi124.stdout:====================================================================================================================== 2023-10-06T01:38:34.974 INFO:teuthology.orchestra.run.smithi124.stdout: Package Arch Version Repository Size 2023-10-06T01:38:34.974 INFO:teuthology.orchestra.run.smithi124.stdout:====================================================================================================================== 2023-10-06T01:38:34.974 INFO:teuthology.orchestra.run.smithi124.stdout:Installing: 2023-10-06T01:38:34.974 INFO:teuthology.orchestra.run.smithi124.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-10-06T01:38:34.974 INFO:teuthology.orchestra.run.smithi124.stdout: ceph x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 6.4 k 2023-10-06T01:38:34.974 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-base x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 5.2 M 2023-10-06T01:38:34.974 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-fuse x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 876 k 2023-10-06T01:38:34.974 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 162 k 2023-10-06T01:38:34.974 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 1.6 M 2023-10-06T01:38:34.975 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-cephadm noarch 2:17.2.6-1296.g736ac5ba.el8 ceph-noarch 121 k 2023-10-06T01:38:34.975 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-dashboard noarch 2:17.2.6-1296.g736ac5ba.el8 ceph-noarch 1.7 M 2023-10-06T01:38:34.975 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-1296.g736ac5ba.el8 ceph-noarch 7.4 M 2023-10-06T01:38:34.975 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-rook noarch 2:17.2.6-1296.g736ac5ba.el8 ceph-noarch 47 k 2023-10-06T01:38:34.975 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-radosgw x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 17 M 2023-10-06T01:38:34.975 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-test x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 52 M 2023-10-06T01:38:34.975 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-volume noarch 2:17.2.6-1296.g736ac5ba.el8 ceph-noarch 267 k 2023-10-06T01:38:34.975 INFO:teuthology.orchestra.run.smithi124.stdout: cephadm noarch 2:17.2.6-1296.g736ac5ba.el8 ceph-noarch 80 k 2023-10-06T01:38:34.975 INFO:teuthology.orchestra.run.smithi124.stdout: cephfs-mirror x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 255 k 2023-10-06T01:38:34.975 INFO:teuthology.orchestra.run.smithi124.stdout: cephfs-top noarch 2:17.2.6-1296.g736ac5ba.el8 ceph-noarch 18 k 2023-10-06T01:38:34.975 INFO:teuthology.orchestra.run.smithi124.stdout: elfutils-libelf-devel x86_64 0.182-3.el8 rhel-8-for-x86_64-baseos-rpms 59 k 2023-10-06T01:38:34.975 INFO:teuthology.orchestra.run.smithi124.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-10-06T01:38:34.976 INFO:teuthology.orchestra.run.smithi124.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-10-06T01:38:34.976 INFO:teuthology.orchestra.run.smithi124.stdout: libcephfs-devel x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 25 k 2023-10-06T01:38:34.976 INFO:teuthology.orchestra.run.smithi124.stdout: libcephfs2 x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 797 k 2023-10-06T01:38:34.976 INFO:teuthology.orchestra.run.smithi124.stdout: librados-devel x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 130 k 2023-10-06T01:38:34.976 INFO:teuthology.orchestra.run.smithi124.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-10-06T01:38:34.976 INFO:teuthology.orchestra.run.smithi124.stdout: libuuid-devel x86_64 2.32.1-27.el8 rhel-8-for-x86_64-baseos-rpms 97 k 2023-10-06T01:38:34.976 INFO:teuthology.orchestra.run.smithi124.stdout: openssl-devel x86_64 1:1.1.1g-15.el8_3 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-10-06T01:38:34.976 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cephfs x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 190 k 2023-10-06T01:38:34.976 INFO:teuthology.orchestra.run.smithi124.stdout: python3-rados x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 371 k 2023-10-06T01:38:34.976 INFO:teuthology.orchestra.run.smithi124.stdout: python3-rbd x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 358 k 2023-10-06T01:38:34.976 INFO:teuthology.orchestra.run.smithi124.stdout: python3-rgw x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 107 k 2023-10-06T01:38:34.976 INFO:teuthology.orchestra.run.smithi124.stdout: rbd-fuse x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 90 k 2023-10-06T01:38:34.977 INFO:teuthology.orchestra.run.smithi124.stdout: rbd-mirror x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 3.5 M 2023-10-06T01:38:34.977 INFO:teuthology.orchestra.run.smithi124.stdout: rbd-nbd x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 173 k 2023-10-06T01:38:34.977 INFO:teuthology.orchestra.run.smithi124.stdout: xfsprogs-devel x86_64 5.0.0-8.el8 rhel-8-for-x86_64-baseos-rpms 70 k 2023-10-06T01:38:34.977 INFO:teuthology.orchestra.run.smithi124.stdout:Upgrading: 2023-10-06T01:38:34.977 INFO:teuthology.orchestra.run.smithi124.stdout: librados2 x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 3.5 M 2023-10-06T01:38:34.978 INFO:teuthology.orchestra.run.smithi124.stdout: librbd1 x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 3.6 M 2023-10-06T01:38:34.978 INFO:teuthology.orchestra.run.smithi124.stdout:Installing dependencies: 2023-10-06T01:38:34.978 INFO:teuthology.orchestra.run.smithi124.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-10-06T01:38:34.978 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-common x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 23 M 2023-10-06T01:38:34.978 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-grafana-dashboards noarch 2:17.2.6-1296.g736ac5ba.el8 ceph-noarch 23 k 2023-10-06T01:38:34.979 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mds x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 2.3 M 2023-10-06T01:38:34.979 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-modules-core noarch 2:17.2.6-1296.g736ac5ba.el8 ceph-noarch 238 k 2023-10-06T01:38:34.979 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mon x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 3.9 M 2023-10-06T01:38:34.979 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-osd x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 17 M 2023-10-06T01:38:34.979 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-prometheus-alerts noarch 2:17.2.6-1296.g736ac5ba.el8 ceph-noarch 14 k 2023-10-06T01:38:34.979 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-selinux x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 24 k 2023-10-06T01:38:34.979 INFO:teuthology.orchestra.run.smithi124.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-10-06T01:38:34.979 INFO:teuthology.orchestra.run.smithi124.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-06T01:38:34.979 INFO:teuthology.orchestra.run.smithi124.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-10-06T01:38:34.979 INFO:teuthology.orchestra.run.smithi124.stdout: keyutils-libs-devel x86_64 1.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-10-06T01:38:34.979 INFO:teuthology.orchestra.run.smithi124.stdout: krb5-devel x86_64 1.18.2-8.3.el8_4 rhel-8-for-x86_64-baseos-rpms 559 k 2023-10-06T01:38:34.980 INFO:teuthology.orchestra.run.smithi124.stdout: libcephsqlite x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 174 k 2023-10-06T01:38:34.980 INFO:teuthology.orchestra.run.smithi124.stdout: libcom_err-devel x86_64 1.45.6-1.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2023-10-06T01:38:34.980 INFO:teuthology.orchestra.run.smithi124.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-10-06T01:38:34.980 INFO:teuthology.orchestra.run.smithi124.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-06T01:38:34.980 INFO:teuthology.orchestra.run.smithi124.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-10-06T01:38:34.980 INFO:teuthology.orchestra.run.smithi124.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-10-06T01:38:34.980 INFO:teuthology.orchestra.run.smithi124.stdout: libradosstriper1 x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 507 k 2023-10-06T01:38:34.980 INFO:teuthology.orchestra.run.smithi124.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-10-06T01:38:34.980 INFO:teuthology.orchestra.run.smithi124.stdout: librgw2 x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 5.6 M 2023-10-06T01:38:34.980 INFO:teuthology.orchestra.run.smithi124.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-10-06T01:38:34.981 INFO:teuthology.orchestra.run.smithi124.stdout: libsepol-devel x86_64 2.9-2.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2023-10-06T01:38:34.981 INFO:teuthology.orchestra.run.smithi124.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-06T01:38:34.981 INFO:teuthology.orchestra.run.smithi124.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-10-06T01:38:34.981 INFO:teuthology.orchestra.run.smithi124.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-10-06T01:38:34.981 INFO:teuthology.orchestra.run.smithi124.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-10-06T01:38:34.981 INFO:teuthology.orchestra.run.smithi124.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-10-06T01:38:34.981 INFO:teuthology.orchestra.run.smithi124.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-10-06T01:38:34.981 INFO:teuthology.orchestra.run.smithi124.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-10-06T01:38:34.981 INFO:teuthology.orchestra.run.smithi124.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-10-06T01:38:34.981 INFO:teuthology.orchestra.run.smithi124.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-10-06T01:38:34.981 INFO:teuthology.orchestra.run.smithi124.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-10-06T01:38:34.982 INFO:teuthology.orchestra.run.smithi124.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-10-06T01:38:34.982 INFO:teuthology.orchestra.run.smithi124.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-06T01:38:34.982 INFO:teuthology.orchestra.run.smithi124.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-06T01:38:34.982 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-06T01:38:34.982 INFO:teuthology.orchestra.run.smithi124.stdout: python3-ceph-argparse x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 45 k 2023-10-06T01:38:34.982 INFO:teuthology.orchestra.run.smithi124.stdout: python3-ceph-common x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 115 k 2023-10-06T01:38:34.982 INFO:teuthology.orchestra.run.smithi124.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-06T01:38:34.982 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-06T01:38:34.982 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-06T01:38:34.982 INFO:teuthology.orchestra.run.smithi124.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-06T01:38:34.982 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-06T01:38:34.983 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-06T01:38:34.983 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-10-06T01:38:34.983 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-10-06T01:38:34.983 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-10-06T01:38:34.983 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-06T01:38:34.983 INFO:teuthology.orchestra.run.smithi124.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-06T01:38:34.983 INFO:teuthology.orchestra.run.smithi124.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-06T01:38:34.983 INFO:teuthology.orchestra.run.smithi124.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-10-06T01:38:34.983 INFO:teuthology.orchestra.run.smithi124.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-10-06T01:38:34.983 INFO:teuthology.orchestra.run.smithi124.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-06T01:38:34.983 INFO:teuthology.orchestra.run.smithi124.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-10-06T01:38:34.983 INFO:teuthology.orchestra.run.smithi124.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-10-06T01:38:34.984 INFO:teuthology.orchestra.run.smithi124.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-10-06T01:38:34.984 INFO:teuthology.orchestra.run.smithi124.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-10-06T01:38:34.984 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-06T01:38:34.984 INFO:teuthology.orchestra.run.smithi124.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-06T01:38:34.984 INFO:teuthology.orchestra.run.smithi124.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-10-06T01:38:34.984 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-10-06T01:38:34.984 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-10-06T01:38:34.984 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-10-06T01:38:34.984 INFO:teuthology.orchestra.run.smithi124.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-06T01:38:34.984 INFO:teuthology.orchestra.run.smithi124.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-06T01:38:34.984 INFO:teuthology.orchestra.run.smithi124.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-06T01:38:34.985 INFO:teuthology.orchestra.run.smithi124.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-06T01:38:34.985 INFO:teuthology.orchestra.run.smithi124.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-10-06T01:38:34.985 INFO:teuthology.orchestra.run.smithi124.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-06T01:38:34.985 INFO:teuthology.orchestra.run.smithi124.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-06T01:38:34.985 INFO:teuthology.orchestra.run.smithi124.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-06T01:38:34.985 INFO:teuthology.orchestra.run.smithi124.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-06T01:38:34.985 INFO:teuthology.orchestra.run.smithi124.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-06T01:38:34.985 INFO:teuthology.orchestra.run.smithi124.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-06T01:38:34.985 INFO:teuthology.orchestra.run.smithi124.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-06T01:38:34.985 INFO:teuthology.orchestra.run.smithi124.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-06T01:38:34.985 INFO:teuthology.orchestra.run.smithi124.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-10-06T01:38:34.985 INFO:teuthology.orchestra.run.smithi124.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-06T01:38:34.986 INFO:teuthology.orchestra.run.smithi124.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-10-06T01:38:34.986 INFO:teuthology.orchestra.run.smithi124.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-10-06T01:38:34.986 INFO:teuthology.orchestra.run.smithi124.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-10-06T01:38:34.986 INFO:teuthology.orchestra.run.smithi124.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2023-10-06T01:38:34.986 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-10-06T01:38:34.986 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction Summary 2023-10-06T01:38:34.986 INFO:teuthology.orchestra.run.smithi124.stdout:====================================================================================================================== 2023-10-06T01:38:34.986 INFO:teuthology.orchestra.run.smithi124.stdout:Install 118 Packages 2023-10-06T01:38:34.986 INFO:teuthology.orchestra.run.smithi124.stdout:Upgrade 2 Packages 2023-10-06T01:38:34.986 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-10-06T01:38:34.987 INFO:teuthology.orchestra.run.smithi124.stdout:Total download size: 201 M 2023-10-06T01:38:34.987 INFO:teuthology.orchestra.run.smithi124.stdout:Downloading Packages: 2023-10-06T01:38:35.138 INFO:teuthology.orchestra.run.smithi124.stdout:(1/120): ceph-17.2.6-1296.g736ac5ba.el8.x86_64. 70 kB/s | 6.4 kB 00:00 2023-10-06T01:38:35.305 INFO:teuthology.orchestra.run.smithi124.stdout:(2/120): ceph-fuse-17.2.6-1296.g736ac5ba.el8.x8 5.1 MB/s | 876 kB 00:00 2023-10-06T01:38:35.327 INFO:teuthology.orchestra.run.smithi124.stdout:(3/120): ceph-immutable-object-cache-17.2.6-129 7.2 MB/s | 162 kB 00:00 2023-10-06T01:38:35.498 INFO:teuthology.orchestra.run.smithi124.stdout:(4/120): ceph-mds-17.2.6-1296.g736ac5ba.el8.x86 13 MB/s | 2.3 MB 00:00 2023-10-06T01:38:35.612 INFO:teuthology.orchestra.run.smithi124.stdout:(5/120): ceph-mgr-17.2.6-1296.g736ac5ba.el8.x86 14 MB/s | 1.6 MB 00:00 2023-10-06T01:38:35.868 INFO:teuthology.orchestra.run.smithi124.stdout:(6/120): ceph-mon-17.2.6-1296.g736ac5ba.el8.x86 15 MB/s | 3.9 MB 00:00 2023-10-06T01:38:36.106 INFO:teuthology.orchestra.run.smithi124.stdout:(7/120): ceph-base-17.2.6-1296.g736ac5ba.el8.x8 4.9 MB/s | 5.2 MB 00:01 2023-10-06T01:38:36.488 INFO:teuthology.orchestra.run.smithi027.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-06T01:38:36.488 INFO:teuthology.orchestra.run.smithi027.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-10-06T01:38:36.500 INFO:teuthology.orchestra.run.smithi027.stdout:Package NetworkManager-1:1.30.0-13.el8_4.x86_64 is already installed. 2023-10-06T01:38:36.500 INFO:teuthology.orchestra.run.smithi027.stdout:Package iproute-5.9.0-4.el8.x86_64 is already installed. 2023-10-06T01:38:36.501 INFO:teuthology.orchestra.run.smithi027.stdout:Package util-linux-2.32.1-27.el8.x86_64 is already installed. 2023-10-06T01:38:36.502 INFO:teuthology.orchestra.run.smithi027.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-10-06T01:38:36.504 INFO:teuthology.orchestra.run.smithi027.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-10-06T01:38:36.506 INFO:teuthology.orchestra.run.smithi027.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2023-10-06T01:38:36.507 INFO:teuthology.orchestra.run.smithi027.stdout:Package xfsprogs-5.0.0-8.el8.x86_64 is already installed. 2023-10-06T01:38:37.030 INFO:teuthology.orchestra.run.smithi027.stdout:Dependencies resolved. 2023-10-06T01:38:37.040 INFO:teuthology.orchestra.run.smithi124.stdout:(8/120): ceph-osd-17.2.6-1296.g736ac5ba.el8.x86 14 MB/s | 17 MB 00:01 2023-10-06T01:38:37.042 INFO:teuthology.orchestra.run.smithi027.stdout:====================================================================================================================== 2023-10-06T01:38:37.042 INFO:teuthology.orchestra.run.smithi027.stdout: Package Arch Version Repository Size 2023-10-06T01:38:37.042 INFO:teuthology.orchestra.run.smithi027.stdout:====================================================================================================================== 2023-10-06T01:38:37.042 INFO:teuthology.orchestra.run.smithi027.stdout:Installing: 2023-10-06T01:38:37.043 INFO:teuthology.orchestra.run.smithi027.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-10-06T01:38:37.043 INFO:teuthology.orchestra.run.smithi027.stdout: ceph x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 6.4 k 2023-10-06T01:38:37.043 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 5.2 M 2023-10-06T01:38:37.043 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 876 k 2023-10-06T01:38:37.043 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 162 k 2023-10-06T01:38:37.043 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 1.6 M 2023-10-06T01:38:37.043 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-cephadm noarch 2:17.2.6-1296.g736ac5ba.el8 ceph-noarch 121 k 2023-10-06T01:38:37.043 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-dashboard noarch 2:17.2.6-1296.g736ac5ba.el8 ceph-noarch 1.7 M 2023-10-06T01:38:37.043 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-1296.g736ac5ba.el8 ceph-noarch 7.4 M 2023-10-06T01:38:37.043 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-rook noarch 2:17.2.6-1296.g736ac5ba.el8 ceph-noarch 47 k 2023-10-06T01:38:37.043 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 17 M 2023-10-06T01:38:37.043 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 52 M 2023-10-06T01:38:37.044 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-volume noarch 2:17.2.6-1296.g736ac5ba.el8 ceph-noarch 267 k 2023-10-06T01:38:37.044 INFO:teuthology.orchestra.run.smithi027.stdout: cephadm noarch 2:17.2.6-1296.g736ac5ba.el8 ceph-noarch 80 k 2023-10-06T01:38:37.044 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-mirror x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 255 k 2023-10-06T01:38:37.044 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-top noarch 2:17.2.6-1296.g736ac5ba.el8 ceph-noarch 18 k 2023-10-06T01:38:37.044 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-libelf-devel x86_64 0.182-3.el8 rhel-8-for-x86_64-baseos-rpms 59 k 2023-10-06T01:38:37.044 INFO:teuthology.orchestra.run.smithi027.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-10-06T01:38:37.044 INFO:teuthology.orchestra.run.smithi027.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-10-06T01:38:37.044 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 25 k 2023-10-06T01:38:37.044 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2 x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 797 k 2023-10-06T01:38:37.044 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 130 k 2023-10-06T01:38:37.044 INFO:teuthology.orchestra.run.smithi027.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-10-06T01:38:37.044 INFO:teuthology.orchestra.run.smithi027.stdout: libuuid-devel x86_64 2.32.1-27.el8 rhel-8-for-x86_64-baseos-rpms 97 k 2023-10-06T01:38:37.045 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-devel x86_64 1:1.1.1g-15.el8_3 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-10-06T01:38:37.045 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 190 k 2023-10-06T01:38:37.045 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 371 k 2023-10-06T01:38:37.045 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 358 k 2023-10-06T01:38:37.045 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 107 k 2023-10-06T01:38:37.045 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 90 k 2023-10-06T01:38:37.045 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 3.5 M 2023-10-06T01:38:37.045 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 173 k 2023-10-06T01:38:37.045 INFO:teuthology.orchestra.run.smithi027.stdout: xfsprogs-devel x86_64 5.0.0-8.el8 rhel-8-for-x86_64-baseos-rpms 70 k 2023-10-06T01:38:37.045 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrading: 2023-10-06T01:38:37.045 INFO:teuthology.orchestra.run.smithi027.stdout: librados2 x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 3.5 M 2023-10-06T01:38:37.045 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1 x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 3.6 M 2023-10-06T01:38:37.046 INFO:teuthology.orchestra.run.smithi027.stdout:Installing dependencies: 2023-10-06T01:38:37.046 INFO:teuthology.orchestra.run.smithi027.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-10-06T01:38:37.046 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 23 M 2023-10-06T01:38:37.046 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-grafana-dashboards noarch 2:17.2.6-1296.g736ac5ba.el8 ceph-noarch 23 k 2023-10-06T01:38:37.046 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 2.3 M 2023-10-06T01:38:37.046 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core noarch 2:17.2.6-1296.g736ac5ba.el8 ceph-noarch 238 k 2023-10-06T01:38:37.046 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 3.9 M 2023-10-06T01:38:37.046 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 17 M 2023-10-06T01:38:37.046 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-prometheus-alerts noarch 2:17.2.6-1296.g736ac5ba.el8 ceph-noarch 14 k 2023-10-06T01:38:37.046 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 24 k 2023-10-06T01:38:37.046 INFO:teuthology.orchestra.run.smithi027.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-10-06T01:38:37.046 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-10-06T01:38:37.047 INFO:teuthology.orchestra.run.smithi027.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-10-06T01:38:37.047 INFO:teuthology.orchestra.run.smithi027.stdout: keyutils-libs-devel x86_64 1.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-10-06T01:38:37.047 INFO:teuthology.orchestra.run.smithi027.stdout: krb5-devel x86_64 1.18.2-8.3.el8_4 rhel-8-for-x86_64-baseos-rpms 559 k 2023-10-06T01:38:37.047 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 174 k 2023-10-06T01:38:37.047 INFO:teuthology.orchestra.run.smithi027.stdout: libcom_err-devel x86_64 1.45.6-1.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2023-10-06T01:38:37.047 INFO:teuthology.orchestra.run.smithi027.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-10-06T01:38:37.047 INFO:teuthology.orchestra.run.smithi027.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-10-06T01:38:37.047 INFO:teuthology.orchestra.run.smithi027.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-10-06T01:38:37.047 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-10-06T01:38:37.047 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1 x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 507 k 2023-10-06T01:38:37.047 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-10-06T01:38:37.048 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2 x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 5.6 M 2023-10-06T01:38:37.048 INFO:teuthology.orchestra.run.smithi027.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-10-06T01:38:37.048 INFO:teuthology.orchestra.run.smithi027.stdout: libsepol-devel x86_64 2.9-2.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2023-10-06T01:38:37.048 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-10-06T01:38:37.048 INFO:teuthology.orchestra.run.smithi027.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-10-06T01:38:37.048 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-10-06T01:38:37.048 INFO:teuthology.orchestra.run.smithi027.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-10-06T01:38:37.048 INFO:teuthology.orchestra.run.smithi027.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-10-06T01:38:37.048 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-10-06T01:38:37.048 INFO:teuthology.orchestra.run.smithi027.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-10-06T01:38:37.048 INFO:teuthology.orchestra.run.smithi027.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-10-06T01:38:37.049 INFO:teuthology.orchestra.run.smithi027.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-10-06T01:38:37.049 INFO:teuthology.orchestra.run.smithi027.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-10-06T01:38:37.049 INFO:teuthology.orchestra.run.smithi027.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-10-06T01:38:37.049 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-10-06T01:38:37.049 INFO:teuthology.orchestra.run.smithi027.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-10-06T01:38:37.049 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-10-06T01:38:37.049 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 45 k 2023-10-06T01:38:37.049 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common x86_64 2:17.2.6-1296.g736ac5ba.el8 ceph 115 k 2023-10-06T01:38:37.049 INFO:teuthology.orchestra.run.smithi027.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-10-06T01:38:37.049 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-10-06T01:38:37.049 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-10-06T01:38:37.050 INFO:teuthology.orchestra.run.smithi027.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-10-06T01:38:37.050 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-10-06T01:38:37.050 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-10-06T01:38:37.050 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-10-06T01:38:37.050 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-10-06T01:38:37.050 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-10-06T01:38:37.050 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-06T01:38:37.050 INFO:teuthology.orchestra.run.smithi027.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-10-06T01:38:37.050 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-10-06T01:38:37.050 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-10-06T01:38:37.051 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-10-06T01:38:37.051 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-10-06T01:38:37.051 INFO:teuthology.orchestra.run.smithi027.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-10-06T01:38:37.051 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-10-06T01:38:37.051 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-10-06T01:38:37.051 INFO:teuthology.orchestra.run.smithi027.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-10-06T01:38:37.051 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-10-06T01:38:37.051 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-10-06T01:38:37.051 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-10-06T01:38:37.051 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-10-06T01:38:37.051 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-10-06T01:38:37.051 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-10-06T01:38:37.052 INFO:teuthology.orchestra.run.smithi027.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-10-06T01:38:37.052 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-10-06T01:38:37.052 INFO:teuthology.orchestra.run.smithi027.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-10-06T01:38:37.052 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-10-06T01:38:37.052 INFO:teuthology.orchestra.run.smithi027.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-10-06T01:38:37.052 INFO:teuthology.orchestra.run.smithi027.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-10-06T01:38:37.052 INFO:teuthology.orchestra.run.smithi027.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-10-06T01:38:37.052 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-10-06T01:38:37.052 INFO:teuthology.orchestra.run.smithi027.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-10-06T01:38:37.053 INFO:teuthology.orchestra.run.smithi027.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-10-06T01:38:37.053 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-10-06T01:38:37.053 INFO:teuthology.orchestra.run.smithi027.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-10-06T01:38:37.053 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-10-06T01:38:37.053 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-10-06T01:38:37.053 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-10-06T01:38:37.053 INFO:teuthology.orchestra.run.smithi027.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-10-06T01:38:37.053 INFO:teuthology.orchestra.run.smithi027.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-10-06T01:38:37.053 INFO:teuthology.orchestra.run.smithi027.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-10-06T01:38:37.053 INFO:teuthology.orchestra.run.smithi027.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2023-10-06T01:38:37.053 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:38:37.054 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction Summary 2023-10-06T01:38:37.054 INFO:teuthology.orchestra.run.smithi027.stdout:====================================================================================================================== 2023-10-06T01:38:37.054 INFO:teuthology.orchestra.run.smithi027.stdout:Install 118 Packages 2023-10-06T01:38:37.054 INFO:teuthology.orchestra.run.smithi027.stdout:Upgrade 2 Packages 2023-10-06T01:38:37.054 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:38:37.055 INFO:teuthology.orchestra.run.smithi027.stdout:Total download size: 201 M 2023-10-06T01:38:37.055 INFO:teuthology.orchestra.run.smithi027.stdout:Downloading Packages: 2023-10-06T01:38:37.064 INFO:teuthology.orchestra.run.smithi124.stdout:(9/120): ceph-selinux-17.2.6-1296.g736ac5ba.el8 1.0 MB/s | 24 kB 00:00 2023-10-06T01:38:37.468 INFO:teuthology.orchestra.run.smithi027.stdout:(1/120): ceph-17.2.6-1296.g736ac5ba.el8.x86_64. 29 kB/s | 6.4 kB 00:00 2023-10-06T01:38:38.145 INFO:teuthology.orchestra.run.smithi027.stdout:(2/120): ceph-fuse-17.2.6-1296.g736ac5ba.el8.x8 1.3 MB/s | 876 kB 00:00 2023-10-06T01:38:38.296 INFO:teuthology.orchestra.run.smithi027.stdout:(3/120): ceph-immutable-object-cache-17.2.6-129 1.0 MB/s | 162 kB 00:00 2023-10-06T01:38:38.996 INFO:teuthology.orchestra.run.smithi124.stdout:(10/120): ceph-radosgw-17.2.6-1296.g736ac5ba.el 5.8 MB/s | 17 MB 00:02 2023-10-06T01:38:39.015 INFO:teuthology.orchestra.run.smithi027.stdout:(4/120): ceph-mds-17.2.6-1296.g736ac5ba.el8.x86 3.2 MB/s | 2.3 MB 00:00 2023-10-06T01:38:39.135 INFO:teuthology.orchestra.run.smithi124.stdout:(11/120): ceph-common-17.2.6-1296.g736ac5ba.el8 5.7 MB/s | 23 MB 00:04 2023-10-06T01:38:39.198 INFO:teuthology.orchestra.run.smithi124.stdout:(12/120): cephfs-mirror-17.2.6-1296.g736ac5ba.e 1.2 MB/s | 255 kB 00:00 2023-10-06T01:38:39.202 INFO:teuthology.orchestra.run.smithi124.stdout:(13/120): libcephfs-devel-17.2.6-1296.g736ac5ba 369 kB/s | 25 kB 00:00 2023-10-06T01:38:39.240 INFO:teuthology.orchestra.run.smithi027.stdout:(5/120): ceph-mgr-17.2.6-1296.g736ac5ba.el8.x86 7.0 MB/s | 1.6 MB 00:00 2023-10-06T01:38:39.287 INFO:teuthology.orchestra.run.smithi124.stdout:(14/120): libcephsqlite-17.2.6-1296.g736ac5ba.e 2.0 MB/s | 174 kB 00:00 2023-10-06T01:38:39.362 INFO:teuthology.orchestra.run.smithi124.stdout:(15/120): libcephfs2-17.2.6-1296.g736ac5ba.el8. 4.8 MB/s | 797 kB 00:00 2023-10-06T01:38:39.374 INFO:teuthology.orchestra.run.smithi124.stdout:(16/120): librados-devel-17.2.6-1296.g736ac5ba. 1.5 MB/s | 130 kB 00:00 2023-10-06T01:38:39.525 INFO:teuthology.orchestra.run.smithi124.stdout:(17/120): libradosstriper1-17.2.6-1296.g736ac5b 3.0 MB/s | 507 kB 00:00 2023-10-06T01:38:39.601 INFO:teuthology.orchestra.run.smithi124.stdout:(18/120): python3-ceph-argparse-17.2.6-1296.g73 590 kB/s | 45 kB 00:00 2023-10-06T01:38:39.679 INFO:teuthology.orchestra.run.smithi124.stdout:(19/120): python3-ceph-common-17.2.6-1296.g736a 1.4 MB/s | 115 kB 00:00 2023-10-06T01:38:39.763 INFO:teuthology.orchestra.run.smithi124.stdout:(20/120): python3-cephfs-17.2.6-1296.g736ac5ba. 2.2 MB/s | 190 kB 00:00 2023-10-06T01:38:39.856 INFO:teuthology.orchestra.run.smithi124.stdout:(21/120): python3-rados-17.2.6-1296.g736ac5ba.e 3.9 MB/s | 371 kB 00:00 2023-10-06T01:38:39.868 INFO:teuthology.orchestra.run.smithi027.stdout:(6/120): ceph-mon-17.2.6-1296.g736ac5ba.el8.x86 6.3 MB/s | 3.9 MB 00:00 2023-10-06T01:38:39.931 INFO:teuthology.orchestra.run.smithi124.stdout:(22/120): python3-rbd-17.2.6-1296.g736ac5ba.el8 4.7 MB/s | 358 kB 00:00 2023-10-06T01:38:40.018 INFO:teuthology.orchestra.run.smithi124.stdout:(23/120): python3-rgw-17.2.6-1296.g736ac5ba.el8 1.2 MB/s | 107 kB 00:00 2023-10-06T01:38:40.109 INFO:teuthology.orchestra.run.smithi124.stdout:(24/120): rbd-fuse-17.2.6-1296.g736ac5ba.el8.x8 989 kB/s | 90 kB 00:00 2023-10-06T01:38:40.583 INFO:teuthology.orchestra.run.smithi124.stdout:(25/120): librgw2-17.2.6-1296.g736ac5ba.el8.x86 4.6 MB/s | 5.6 MB 00:01 2023-10-06T01:38:40.678 INFO:teuthology.orchestra.run.smithi124.stdout:(26/120): rbd-nbd-17.2.6-1296.g736ac5ba.el8.x86 1.8 MB/s | 173 kB 00:00 2023-10-06T01:38:40.763 INFO:teuthology.orchestra.run.smithi124.stdout:(27/120): ceph-grafana-dashboards-17.2.6-1296.g 270 kB/s | 23 kB 00:00 2023-10-06T01:38:40.857 INFO:teuthology.orchestra.run.smithi124.stdout:(28/120): ceph-mgr-cephadm-17.2.6-1296.g736ac5b 1.3 MB/s | 121 kB 00:00 2023-10-06T01:38:40.927 INFO:teuthology.orchestra.run.smithi124.stdout:(29/120): rbd-mirror-17.2.6-1296.g736ac5ba.el8. 4.2 MB/s | 3.5 MB 00:00 2023-10-06T01:38:41.436 INFO:teuthology.orchestra.run.smithi124.stdout:(30/120): ceph-mgr-dashboard-17.2.6-1296.g736ac 2.9 MB/s | 1.7 MB 00:00 2023-10-06T01:38:41.538 INFO:teuthology.orchestra.run.smithi124.stdout:(31/120): ceph-mgr-modules-core-17.2.6-1296.g73 2.3 MB/s | 238 kB 00:00 2023-10-06T01:38:41.636 INFO:teuthology.orchestra.run.smithi124.stdout:(32/120): ceph-mgr-rook-17.2.6-1296.g736ac5ba.e 484 kB/s | 47 kB 00:00 2023-10-06T01:38:41.720 INFO:teuthology.orchestra.run.smithi124.stdout:(33/120): ceph-prometheus-alerts-17.2.6-1296.g7 172 kB/s | 14 kB 00:00 2023-10-06T01:38:41.823 INFO:teuthology.orchestra.run.smithi124.stdout:(34/120): ceph-volume-17.2.6-1296.g736ac5ba.el8 2.5 MB/s | 267 kB 00:00 2023-10-06T01:38:41.912 INFO:teuthology.orchestra.run.smithi124.stdout:(35/120): cephadm-17.2.6-1296.g736ac5ba.el8.noa 913 kB/s | 80 kB 00:00 2023-10-06T01:38:41.983 INFO:teuthology.orchestra.run.smithi124.stdout:(36/120): cephfs-top-17.2.6-1296.g736ac5ba.el8. 255 kB/s | 18 kB 00:00 2023-10-06T01:38:42.040 INFO:teuthology.orchestra.run.smithi027.stdout:(7/120): ceph-osd-17.2.6-1296.g736ac5ba.el8.x86 7.7 MB/s | 17 MB 00:02 2023-10-06T01:38:42.374 INFO:teuthology.orchestra.run.smithi124.stdout:(37/120): ceph-test-17.2.6-1296.g736ac5ba.el8.x 9.7 MB/s | 52 MB 00:05 2023-10-06T01:38:42.465 INFO:teuthology.orchestra.run.smithi124.stdout:(38/120): ceph-mgr-diskprediction-local-17.2.6- 4.8 MB/s | 7.4 MB 00:01 2023-10-06T01:38:42.564 INFO:teuthology.orchestra.run.smithi027.stdout:(8/120): ceph-common-17.2.6-1296.g736ac5ba.el8. 4.4 MB/s | 23 MB 00:05 2023-10-06T01:38:42.596 INFO:teuthology.orchestra.run.smithi027.stdout:(9/120): ceph-selinux-17.2.6-1296.g736ac5ba.el8 772 kB/s | 24 kB 00:00 2023-10-06T01:38:42.879 INFO:teuthology.orchestra.run.smithi027.stdout:(10/120): ceph-base-17.2.6-1296.g736ac5ba.el8.x 939 kB/s | 5.2 MB 00:05 2023-10-06T01:38:43.143 INFO:teuthology.orchestra.run.smithi027.stdout:(11/120): ceph-radosgw-17.2.6-1296.g736ac5ba.el 15 MB/s | 17 MB 00:01 2023-10-06T01:38:43.145 INFO:teuthology.orchestra.run.smithi027.stdout:(12/120): cephfs-mirror-17.2.6-1296.g736ac5ba.e 959 kB/s | 255 kB 00:00 2023-10-06T01:38:43.165 INFO:teuthology.orchestra.run.smithi027.stdout:(13/120): libcephfs-devel-17.2.6-1296.g736ac5ba 1.1 MB/s | 25 kB 00:00 2023-10-06T01:38:43.193 INFO:teuthology.orchestra.run.smithi027.stdout:(14/120): libcephsqlite-17.2.6-1296.g736ac5ba.e 6.2 MB/s | 174 kB 00:00 2023-10-06T01:38:43.215 INFO:teuthology.orchestra.run.smithi027.stdout:(15/120): librados-devel-17.2.6-1296.g736ac5ba. 5.7 MB/s | 130 kB 00:00 2023-10-06T01:38:43.246 INFO:teuthology.orchestra.run.smithi027.stdout:(16/120): libradosstriper1-17.2.6-1296.g736ac5b 16 MB/s | 507 kB 00:00 2023-10-06T01:38:43.443 INFO:teuthology.orchestra.run.smithi124.stdout:(39/120): liboath-2.6.2-3.el8.x86_64.rpm 55 kB/s | 59 kB 00:01 2023-10-06T01:38:43.565 INFO:teuthology.orchestra.run.smithi124.stdout:(40/120): libunwind-1.3.1-3.el8.x86_64.rpm 68 kB/s | 75 kB 00:01 2023-10-06T01:38:43.573 INFO:teuthology.orchestra.run.smithi027.stdout:(17/120): librgw2-17.2.6-1296.g736ac5ba.el8.x86 17 MB/s | 5.6 MB 00:00 2023-10-06T01:38:43.591 INFO:teuthology.orchestra.run.smithi027.stdout:(18/120): libcephfs2-17.2.6-1296.g736ac5ba.el8. 1.7 MB/s | 797 kB 00:00 2023-10-06T01:38:43.597 INFO:teuthology.orchestra.run.smithi027.stdout:(19/120): python3-ceph-argparse-17.2.6-1296.g73 1.8 MB/s | 45 kB 00:00 2023-10-06T01:38:43.624 INFO:teuthology.orchestra.run.smithi027.stdout:(20/120): python3-cephfs-17.2.6-1296.g736ac5ba. 6.9 MB/s | 190 kB 00:00 2023-10-06T01:38:43.650 INFO:teuthology.orchestra.run.smithi027.stdout:(21/120): python3-rados-17.2.6-1296.g736ac5ba.e 14 MB/s | 371 kB 00:00 2023-10-06T01:38:43.674 INFO:teuthology.orchestra.run.smithi027.stdout:(22/120): python3-rbd-17.2.6-1296.g736ac5ba.el8 15 MB/s | 358 kB 00:00 2023-10-06T01:38:43.693 INFO:teuthology.orchestra.run.smithi027.stdout:(23/120): python3-ceph-common-17.2.6-1296.g736a 1.1 MB/s | 115 kB 00:00 2023-10-06T01:38:43.696 INFO:teuthology.orchestra.run.smithi027.stdout:(24/120): python3-rgw-17.2.6-1296.g736ac5ba.el8 4.8 MB/s | 107 kB 00:00 2023-10-06T01:38:43.739 INFO:teuthology.orchestra.run.smithi027.stdout:(25/120): rbd-fuse-17.2.6-1296.g736ac5ba.el8.x8 1.9 MB/s | 90 kB 00:00 2023-10-06T01:38:43.846 INFO:teuthology.orchestra.run.smithi124.stdout:(41/120): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 155 kB/s | 44 kB 00:00 2023-10-06T01:38:43.865 INFO:teuthology.orchestra.run.smithi027.stdout:(26/120): rbd-nbd-17.2.6-1296.g736ac5ba.el8.x86 1.3 MB/s | 173 kB 00:00 2023-10-06T01:38:43.901 INFO:teuthology.orchestra.run.smithi027.stdout:(27/120): rbd-mirror-17.2.6-1296.g736ac5ba.el8. 17 MB/s | 3.5 MB 00:00 2023-10-06T01:38:43.902 INFO:teuthology.orchestra.run.smithi124.stdout:(42/120): gperftools-libs-2.7-9.el8.x86_64.rpm 160 kB/s | 306 kB 00:01 2023-10-06T01:38:43.917 INFO:teuthology.orchestra.run.smithi027.stdout:(28/120): ceph-grafana-dashboards-17.2.6-1296.g 438 kB/s | 23 kB 00:00 2023-10-06T01:38:43.926 INFO:teuthology.orchestra.run.smithi027.stdout:(29/120): ceph-mgr-cephadm-17.2.6-1296.g736ac5b 4.9 MB/s | 121 kB 00:00 2023-10-06T01:38:44.128 INFO:teuthology.orchestra.run.smithi124.stdout:(43/120): python3-cachetools-3.1.1-4.el8.noarch 146 kB/s | 33 kB 00:00 2023-10-06T01:38:44.300 INFO:teuthology.orchestra.run.smithi124.stdout:(44/120): python3-asyncssh-2.7.0-2.el8.noarch.r 555 kB/s | 476 kB 00:00 2023-10-06T01:38:44.358 INFO:teuthology.orchestra.run.smithi124.stdout:(45/120): python3-beautifulsoup4-4.6.3-2.el8.1. 362 kB/s | 185 kB 00:00 2023-10-06T01:38:44.358 INFO:teuthology.orchestra.run.smithi124.stdout:(46/120): python3-certifi-2018.10.15-7.el8.noar 71 kB/s | 16 kB 00:00 2023-10-06T01:38:44.364 INFO:teuthology.orchestra.run.smithi027.stdout:(30/120): ceph-mgr-diskprediction-local-17.2.6- 17 MB/s | 7.4 MB 00:00 2023-10-06T01:38:44.397 INFO:teuthology.orchestra.run.smithi027.stdout:(31/120): ceph-mgr-modules-core-17.2.6-1296.g73 7.5 MB/s | 238 kB 00:00 2023-10-06T01:38:44.416 INFO:teuthology.orchestra.run.smithi027.stdout:(32/120): ceph-mgr-rook-17.2.6-1296.g736ac5ba.e 2.2 MB/s | 47 kB 00:00 2023-10-06T01:38:44.437 INFO:teuthology.orchestra.run.smithi027.stdout:(33/120): ceph-prometheus-alerts-17.2.6-1296.g7 693 kB/s | 14 kB 00:00 2023-10-06T01:38:44.460 INFO:teuthology.orchestra.run.smithi027.stdout:(34/120): ceph-volume-17.2.6-1296.g736ac5ba.el8 11 MB/s | 267 kB 00:00 2023-10-06T01:38:44.482 INFO:teuthology.orchestra.run.smithi027.stdout:(35/120): cephadm-17.2.6-1296.g736ac5ba.el8.noa 3.6 MB/s | 80 kB 00:00 2023-10-06T01:38:44.504 INFO:teuthology.orchestra.run.smithi027.stdout:(36/120): cephfs-top-17.2.6-1296.g736ac5ba.el8. 860 kB/s | 18 kB 00:00 2023-10-06T01:38:44.554 INFO:teuthology.orchestra.run.smithi124.stdout:(47/120): python3-cheroot-8.5.2-1.el8.noarch.rp 679 kB/s | 173 kB 00:00 2023-10-06T01:38:44.622 INFO:teuthology.orchestra.run.smithi124.stdout:(48/120): python3-google-auth-1.1.1-10.el8.noar 308 kB/s | 81 kB 00:00 2023-10-06T01:38:44.761 INFO:teuthology.orchestra.run.smithi027.stdout:(37/120): ceph-mgr-dashboard-17.2.6-1296.g736ac 2.0 MB/s | 1.7 MB 00:00 2023-10-06T01:38:44.763 INFO:teuthology.orchestra.run.smithi124.stdout:(49/120): python3-jaraco-6.2-6.el8.noarch.rpm 53 kB/s | 11 kB 00:00 2023-10-06T01:38:44.773 INFO:teuthology.orchestra.run.smithi027.stdout:(38/120): gperftools-libs-2.7-9.el8.x86_64.rpm 1.1 MB/s | 306 kB 00:00 2023-10-06T01:38:44.803 INFO:teuthology.orchestra.run.smithi124.stdout:(50/120): python3-cherrypy-18.4.0-1.el8.noarch. 864 kB/s | 384 kB 00:00 2023-10-06T01:38:44.814 INFO:teuthology.orchestra.run.smithi027.stdout:(39/120): libunwind-1.3.1-3.el8.x86_64.rpm 1.8 MB/s | 75 kB 00:00 2023-10-06T01:38:44.836 INFO:teuthology.orchestra.run.smithi124.stdout:(51/120): python3-jaraco-functools-2.0-4.el8.no 82 kB/s | 18 kB 00:00 2023-10-06T01:38:44.898 INFO:teuthology.orchestra.run.smithi027.stdout:(40/120): python3-asyncssh-2.7.0-2.el8.noarch.r 5.6 MB/s | 476 kB 00:00 2023-10-06T01:38:44.924 INFO:teuthology.orchestra.run.smithi027.stdout:(41/120): liboath-2.6.2-3.el8.x86_64.rpm 361 kB/s | 59 kB 00:00 2023-10-06T01:38:44.939 INFO:teuthology.orchestra.run.smithi027.stdout:(42/120): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 1.0 MB/s | 44 kB 00:00 2023-10-06T01:38:44.995 INFO:teuthology.orchestra.run.smithi027.stdout:(43/120): python3-cachetools-3.1.1-4.el8.noarch 600 kB/s | 33 kB 00:00 2023-10-06T01:38:45.028 INFO:teuthology.orchestra.run.smithi124.stdout:(52/120): python3-logutils-0.3.5-11.el8.noarch. 217 kB/s | 49 kB 00:00 2023-10-06T01:38:45.045 INFO:teuthology.orchestra.run.smithi027.stdout:(44/120): python3-certifi-2018.10.15-7.el8.noar 324 kB/s | 16 kB 00:00 2023-10-06T01:38:45.046 INFO:teuthology.orchestra.run.smithi027.stdout:(45/120): python3-beautifulsoup4-4.6.3-2.el8.1. 1.5 MB/s | 185 kB 00:00 2023-10-06T01:38:45.080 INFO:teuthology.orchestra.run.smithi124.stdout:(53/120): python3-more-itertools-7.2.0-3.el8.no 240 kB/s | 59 kB 00:00 2023-10-06T01:38:45.094 INFO:teuthology.orchestra.run.smithi027.stdout:(46/120): python3-cheroot-8.5.2-1.el8.noarch.rp 3.5 MB/s | 173 kB 00:00 2023-10-06T01:38:45.144 INFO:teuthology.orchestra.run.smithi027.stdout:(47/120): python3-cherrypy-18.4.0-1.el8.noarch. 3.8 MB/s | 384 kB 00:00 2023-10-06T01:38:45.147 INFO:teuthology.orchestra.run.smithi027.stdout:(48/120): python3-google-auth-1.1.1-10.el8.noar 1.5 MB/s | 81 kB 00:00 2023-10-06T01:38:45.193 INFO:teuthology.orchestra.run.smithi027.stdout:(49/120): python3-jaraco-functools-2.0-4.el8.no 384 kB/s | 18 kB 00:00 2023-10-06T01:38:45.194 INFO:teuthology.orchestra.run.smithi027.stdout:(50/120): python3-jaraco-6.2-6.el8.noarch.rpm 222 kB/s | 11 kB 00:00 2023-10-06T01:38:45.243 INFO:teuthology.orchestra.run.smithi027.stdout:(51/120): python3-logutils-0.3.5-11.el8.noarch. 1.0 MB/s | 49 kB 00:00 2023-10-06T01:38:45.253 INFO:teuthology.orchestra.run.smithi124.stdout:(54/120): python3-natsort-7.1.1-2.el8.noarch.rp 267 kB/s | 60 kB 00:00 2023-10-06T01:38:45.294 INFO:teuthology.orchestra.run.smithi027.stdout:(52/120): python3-more-itertools-7.2.0-3.el8.no 1.1 MB/s | 59 kB 00:00 2023-10-06T01:38:45.300 INFO:teuthology.orchestra.run.smithi027.stdout:(53/120): python3-kubernetes-11.0.0-6.el8.noarc 11 MB/s | 1.2 MB 00:00 2023-10-06T01:38:45.336 INFO:teuthology.orchestra.run.smithi027.stdout:(54/120): python3-natsort-7.1.1-2.el8.noarch.rp 1.4 MB/s | 60 kB 00:00 2023-10-06T01:38:45.349 INFO:teuthology.orchestra.run.smithi027.stdout:(55/120): python3-pecan-1.3.2-9.el8.noarch.rpm 5.7 MB/s | 283 kB 00:00 2023-10-06T01:38:45.375 INFO:teuthology.orchestra.run.smithi124.stdout:(55/120): python3-kubernetes-11.0.0-6.el8.noarc 1.9 MB/s | 1.2 MB 00:00 2023-10-06T01:38:45.378 INFO:teuthology.orchestra.run.smithi027.stdout:(56/120): python3-portend-2.6-1.el8.noarch.rpm 371 kB/s | 16 kB 00:00 2023-10-06T01:38:45.390 INFO:teuthology.orchestra.run.smithi027.stdout:(57/120): python3-repoze-lru-0.7-6.el8.noarch.r 825 kB/s | 34 kB 00:00 2023-10-06T01:38:45.422 INFO:teuthology.orchestra.run.smithi027.stdout:(58/120): python3-routes-2.4.1-12.el8.noarch.rp 4.4 MB/s | 196 kB 00:00 2023-10-06T01:38:45.438 INFO:teuthology.orchestra.run.smithi027.stdout:(59/120): python3-rsa-4.9-2.el8.noarch.rpm 1.3 MB/s | 64 kB 00:00 2023-10-06T01:38:45.460 INFO:teuthology.orchestra.run.smithi124.stdout:(56/120): python3-pecan-1.3.2-9.el8.noarch.rpm 744 kB/s | 283 kB 00:00 2023-10-06T01:38:45.470 INFO:teuthology.orchestra.run.smithi124.stdout:(57/120): python3-portend-2.6-1.el8.noarch.rpm 72 kB/s | 16 kB 00:00 2023-10-06T01:38:45.472 INFO:teuthology.orchestra.run.smithi027.stdout:(60/120): python3-simplegeneric-0.8.1-17.el8.no 369 kB/s | 19 kB 00:00 2023-10-06T01:38:45.479 INFO:teuthology.orchestra.run.smithi027.stdout:(61/120): python3-singledispatch-3.4.0.3-18.el8 579 kB/s | 24 kB 00:00 2023-10-06T01:38:45.522 INFO:teuthology.orchestra.run.smithi027.stdout:(62/120): python3-tempora-1.14.1-5.el8.noarch.r 598 kB/s | 29 kB 00:00 2023-10-06T01:38:45.528 INFO:teuthology.orchestra.run.smithi027.stdout:(63/120): python3-trustme-0.6.0-4.el8.noarch.rp 562 kB/s | 27 kB 00:00 2023-10-06T01:38:45.580 INFO:teuthology.orchestra.run.smithi027.stdout:(64/120): python3-waitress-1.4.3-1.el8.noarch.r 4.3 MB/s | 255 kB 00:00 2023-10-06T01:38:45.582 INFO:teuthology.orchestra.run.smithi027.stdout:(65/120): python3-webob-1.8.5-1.el8.1.noarch.rp 4.6 MB/s | 251 kB 00:00 2023-10-06T01:38:45.602 INFO:teuthology.orchestra.run.smithi124.stdout:(58/120): python3-repoze-lru-0.7-6.el8.noarch.r 148 kB/s | 34 kB 00:00 2023-10-06T01:38:45.625 INFO:teuthology.orchestra.run.smithi027.stdout:(66/120): python3-websocket-client-0.56.0-5.el8 1.4 MB/s | 61 kB 00:00 2023-10-06T01:38:45.625 INFO:teuthology.orchestra.run.smithi027.stdout:(67/120): python3-webtest-2.0.33-1.el8.noarch.r 1.9 MB/s | 85 kB 00:00 2023-10-06T01:38:45.674 INFO:teuthology.orchestra.run.smithi027.stdout:(68/120): python3-zc-lockfile-2.0-2.el8.noarch. 470 kB/s | 23 kB 00:00 2023-10-06T01:38:45.678 INFO:teuthology.orchestra.run.smithi027.stdout:(69/120): xmlstarlet-1.6.1-20.el8.x86_64.rpm 18 MB/s | 70 kB 00:00 2023-10-06T01:38:45.702 INFO:teuthology.orchestra.run.smithi027.stdout:(70/120): thrift-0.13.0-2.el8.x86_64.rpm 23 MB/s | 1.7 MB 00:00 2023-10-06T01:38:45.709 INFO:teuthology.orchestra.run.smithi124.stdout:(59/120): python3-rsa-4.9-2.el8.noarch.rpm 268 kB/s | 64 kB 00:00 2023-10-06T01:38:45.719 INFO:teuthology.orchestra.run.smithi027.stdout:(71/120): libaio-devel-0.3.112-1.el8.x86_64.rpm 459 kB/s | 19 kB 00:00 2023-10-06T01:38:45.735 INFO:teuthology.orchestra.run.smithi027.stdout:(72/120): python3-jwt-1.6.1-2.el8.noarch.rpm 1.3 MB/s | 43 kB 00:00 2023-10-06T01:38:45.737 INFO:teuthology.orchestra.run.smithi027.stdout:(73/120): python3-oauthlib-2.1.0-1.el8.noarch.r 8.5 MB/s | 155 kB 00:00 2023-10-06T01:38:45.754 INFO:teuthology.orchestra.run.smithi027.stdout:(74/120): pcre2-utf32-10.32-2.el8.x86_64.rpm 12 MB/s | 220 kB 00:00 2023-10-06T01:38:45.754 INFO:teuthology.orchestra.run.smithi027.stdout:(75/120): keyutils-libs-devel-1.5.10-6.el8.x86_ 2.7 MB/s | 48 kB 00:00 2023-10-06T01:38:45.768 INFO:teuthology.orchestra.run.smithi124.stdout:(60/120): python3-routes-2.4.1-12.el8.noarch.rp 639 kB/s | 196 kB 00:00 2023-10-06T01:38:45.770 INFO:teuthology.orchestra.run.smithi027.stdout:(76/120): python3-requests-oauthlib-1.0.0-1.el8 2.7 MB/s | 43 kB 00:00 2023-10-06T01:38:45.772 INFO:teuthology.orchestra.run.smithi027.stdout:(77/120): pcre2-utf16-10.32-2.el8.x86_64.rpm 12 MB/s | 229 kB 00:00 2023-10-06T01:38:45.786 INFO:teuthology.orchestra.run.smithi027.stdout:(78/120): libverto-devel-0.3.0-5.el8.x86_64.rpm 1.3 MB/s | 18 kB 00:00 2023-10-06T01:38:45.809 INFO:teuthology.orchestra.run.smithi027.stdout:(79/120): pcre2-devel-10.32-2.el8.x86_64.rpm 26 MB/s | 605 kB 00:00 2023-10-06T01:38:45.810 INFO:teuthology.orchestra.run.smithi124.stdout:(61/120): python3-simplegeneric-0.8.1-17.el8.no 90 kB/s | 19 kB 00:00 2023-10-06T01:38:45.819 INFO:teuthology.orchestra.run.smithi027.stdout:(80/120): openssl-devel-1.1.1g-15.el8_3.x86_64. 47 MB/s | 2.3 MB 00:00 2023-10-06T01:38:45.823 INFO:teuthology.orchestra.run.smithi027.stdout:(81/120): fuse-2.9.7-12.el8.x86_64.rpm 5.6 MB/s | 83 kB 00:00 2023-10-06T01:38:45.833 INFO:teuthology.orchestra.run.smithi027.stdout:(82/120): elfutils-libelf-devel-0.182-3.el8.x86 4.2 MB/s | 59 kB 00:00 2023-10-06T01:38:45.838 INFO:teuthology.orchestra.run.smithi027.stdout:(83/120): libquadmath-8.4.1-1.el8.x86_64.rpm 11 MB/s | 169 kB 00:00 2023-10-06T01:38:45.847 INFO:teuthology.orchestra.run.smithi027.stdout:(84/120): libcom_err-devel-1.45.6-1.el8.x86_64. 2.7 MB/s | 38 kB 00:00 2023-10-06T01:38:45.853 INFO:teuthology.orchestra.run.smithi027.stdout:(85/120): libsepol-devel-2.9-2.el8.x86_64.rpm 5.8 MB/s | 86 kB 00:00 2023-10-06T01:38:45.863 INFO:teuthology.orchestra.run.smithi027.stdout:(86/120): zlib-devel-1.2.11-17.el8.x86_64.rpm 3.8 MB/s | 58 kB 00:00 2023-10-06T01:38:45.870 INFO:teuthology.orchestra.run.smithi027.stdout:(87/120): librabbitmq-0.9.0-3.el8.x86_64.rpm 2.6 MB/s | 47 kB 00:00 2023-10-06T01:38:45.883 INFO:teuthology.orchestra.run.smithi027.stdout:(88/120): krb5-devel-1.18.2-8.3.el8_4.x86_64.rp 27 MB/s | 559 kB 00:00 2023-10-06T01:38:45.892 INFO:teuthology.orchestra.run.smithi027.stdout:(89/120): libgfortran-8.4.1-1.el8.x86_64.rpm 30 MB/s | 639 kB 00:00 2023-10-06T01:38:45.896 INFO:teuthology.orchestra.run.smithi027.stdout:(90/120): xfsprogs-devel-5.0.0-8.el8.x86_64.rpm 5.2 MB/s | 70 kB 00:00 2023-10-06T01:38:45.906 INFO:teuthology.orchestra.run.smithi027.stdout:(91/120): libselinux-devel-2.9-5.el8.x86_64.rpm 13 MB/s | 200 kB 00:00 2023-10-06T01:38:45.909 INFO:teuthology.orchestra.run.smithi027.stdout:(92/120): libuuid-devel-2.32.1-27.el8.x86_64.rp 7.4 MB/s | 97 kB 00:00 2023-10-06T01:38:45.922 INFO:teuthology.orchestra.run.smithi027.stdout:(93/120): python3-prettytable-0.7.2-14.el8.noar 3.5 MB/s | 44 kB 00:00 2023-10-06T01:38:45.929 INFO:teuthology.orchestra.run.smithi124.stdout:(62/120): python3-singledispatch-3.4.0.3-18.el8 109 kB/s | 24 kB 00:00 2023-10-06T01:38:45.934 INFO:teuthology.orchestra.run.smithi027.stdout:(94/120): python3-jsonpatch-1.21-2.el8.noarch.r 2.2 MB/s | 27 kB 00:00 2023-10-06T01:38:45.948 INFO:teuthology.orchestra.run.smithi027.stdout:(95/120): jq-1.5-12.el8.x86_64.rpm 11 MB/s | 161 kB 00:00 2023-10-06T01:38:45.965 INFO:teuthology.orchestra.run.smithi027.stdout:(96/120): flex-2.6.1-9.el8.x86_64.rpm 19 MB/s | 320 kB 00:00 2023-10-06T01:38:45.984 INFO:teuthology.orchestra.run.smithi027.stdout:(97/120): python3-werkzeug-0.12.2-4.el8.noarch. 24 MB/s | 457 kB 00:00 2023-10-06T01:38:45.998 INFO:teuthology.orchestra.run.smithi124.stdout:(63/120): python3-tempora-1.14.1-5.el8.noarch.r 127 kB/s | 29 kB 00:00 2023-10-06T01:38:46.034 INFO:teuthology.orchestra.run.smithi124.stdout:(64/120): python3-trustme-0.6.0-4.el8.noarch.rp 123 kB/s | 27 kB 00:00 2023-10-06T01:38:46.075 INFO:teuthology.orchestra.run.smithi027.stdout:(98/120): python3-babel-2.5.1-5.el8.noarch.rpm 53 MB/s | 4.8 MB 00:00 2023-10-06T01:38:46.088 INFO:teuthology.orchestra.run.smithi027.stdout:(99/120): python3-markupsafe-0.23-19.el8.x86_64 3.0 MB/s | 39 kB 00:00 2023-10-06T01:38:46.110 INFO:teuthology.orchestra.run.smithi027.stdout:(100/120): bison-3.0.4-10.el8.x86_64.rpm 31 MB/s | 688 kB 00:00 2023-10-06T01:38:46.207 INFO:teuthology.orchestra.run.smithi124.stdout:(65/120): python3-waitress-1.4.3-1.el8.noarch.r 919 kB/s | 255 kB 00:00 2023-10-06T01:38:46.227 INFO:teuthology.orchestra.run.smithi027.stdout:(101/120): python3-scipy-1.0.0-20.module+el8.1. 45 MB/s | 15 MB 00:00 2023-10-06T01:38:46.233 INFO:teuthology.orchestra.run.smithi027.stdout:(102/120): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.1 MB/s | 259 kB 00:00 2023-10-06T01:38:46.243 INFO:teuthology.orchestra.run.smithi027.stdout:(103/120): python3-numpy-f2py-1.14.3-9.el8.x86_ 14 MB/s | 225 kB 00:00 2023-10-06T01:38:46.247 INFO:teuthology.orchestra.run.smithi027.stdout:(104/120): python3-pyasn1-0.3.7-6.el8.noarch.rp 8.2 MB/s | 126 kB 00:00 2023-10-06T01:38:46.264 INFO:teuthology.orchestra.run.smithi027.stdout:(105/120): oniguruma-6.8.2-2.el8.x86_64.rpm 11 MB/s | 187 kB 00:00 2023-10-06T01:38:46.275 INFO:teuthology.orchestra.run.smithi124.stdout:(66/120): python3-websocket-client-0.56.0-5.el8 255 kB/s | 61 kB 00:00 2023-10-06T01:38:46.286 INFO:teuthology.orchestra.run.smithi027.stdout:(106/120): python3-jinja2-2.10.1-2.el8_0.noarch 24 MB/s | 538 kB 00:00 2023-10-06T01:38:46.303 INFO:teuthology.orchestra.run.smithi027.stdout:(107/120): socat-1.7.3.3-2.el8.x86_64.rpm 18 MB/s | 302 kB 00:00 2023-10-06T01:38:46.315 INFO:teuthology.orchestra.run.smithi027.stdout:(108/120): python3-numpy-1.14.3-9.el8.x86_64.rp 52 MB/s | 3.7 MB 00:00 2023-10-06T01:38:46.317 INFO:teuthology.orchestra.run.smithi027.stdout:(109/120): python3-mako-1.0.6-13.el8.noarch.rpm 11 MB/s | 157 kB 00:00 2023-10-06T01:38:46.328 INFO:teuthology.orchestra.run.smithi027.stdout:(110/120): python3-pyasn1-modules-0.3.7-6.el8.n 8.4 MB/s | 110 kB 00:00 2023-10-06T01:38:46.334 INFO:teuthology.orchestra.run.smithi027.stdout:(111/120): librdkafka-0.11.4-1.el8.x86_64.rpm 21 MB/s | 353 kB 00:00 2023-10-06T01:38:46.340 INFO:teuthology.orchestra.run.smithi027.stdout:(112/120): python3-jsonpointer-1.10-11.el8.noar 1.6 MB/s | 20 kB 00:00 2023-10-06T01:38:46.340 INFO:teuthology.orchestra.run.smithi124.stdout:(67/120): python3-webob-1.8.5-1.el8.1.noarch.rp 735 kB/s | 251 kB 00:00 2023-10-06T01:38:46.356 INFO:teuthology.orchestra.run.smithi027.stdout:(113/120): libtool-2.4.6-25.el8.x86_64.rpm 32 MB/s | 709 kB 00:00 2023-10-06T01:38:46.356 INFO:teuthology.orchestra.run.smithi027.stdout:(114/120): python3-pytz-2017.2-9.el8.noarch.rpm 3.3 MB/s | 54 kB 00:00 2023-10-06T01:38:46.451 INFO:teuthology.orchestra.run.smithi124.stdout:(68/120): python3-webtest-2.0.33-1.el8.noarch.r 350 kB/s | 85 kB 00:00 2023-10-06T01:38:46.456 INFO:teuthology.orchestra.run.smithi124.stdout:(69/120): xmlstarlet-1.6.1-20.el8.x86_64.rpm 15 MB/s | 70 kB 00:00 2023-10-06T01:38:46.483 INFO:teuthology.orchestra.run.smithi124.stdout:(70/120): libaio-devel-0.3.112-1.el8.x86_64.rpm 687 kB/s | 19 kB 00:00 2023-10-06T01:38:46.495 INFO:teuthology.orchestra.run.smithi124.stdout:(71/120): python3-jwt-1.6.1-2.el8.noarch.rpm 3.6 MB/s | 43 kB 00:00 2023-10-06T01:38:46.508 INFO:teuthology.orchestra.run.smithi124.stdout:(72/120): python3-oauthlib-2.1.0-1.el8.noarch.r 12 MB/s | 155 kB 00:00 2023-10-06T01:38:46.511 INFO:teuthology.orchestra.run.smithi124.stdout:(73/120): python3-zc-lockfile-2.0-2.el8.noarch. 97 kB/s | 23 kB 00:00 2023-10-06T01:38:46.527 INFO:teuthology.orchestra.run.smithi124.stdout:(74/120): pcre2-utf32-10.32-2.el8.x86_64.rpm 11 MB/s | 220 kB 00:00 2023-10-06T01:38:46.538 INFO:teuthology.orchestra.run.smithi124.stdout:(75/120): keyutils-libs-devel-1.5.10-6.el8.x86_ 1.8 MB/s | 48 kB 00:00 2023-10-06T01:38:46.544 INFO:teuthology.orchestra.run.smithi124.stdout:(76/120): pcre2-utf16-10.32-2.el8.x86_64.rpm 14 MB/s | 229 kB 00:00 2023-10-06T01:38:46.551 INFO:teuthology.orchestra.run.smithi124.stdout:(77/120): python3-requests-oauthlib-1.0.0-1.el8 3.3 MB/s | 43 kB 00:00 2023-10-06T01:38:46.564 INFO:teuthology.orchestra.run.smithi124.stdout:(78/120): libverto-devel-0.3.0-5.el8.x86_64.rpm 1.3 MB/s | 18 kB 00:00 2023-10-06T01:38:46.587 INFO:teuthology.orchestra.run.smithi124.stdout:(79/120): openssl-devel-1.1.1g-15.el8_3.x86_64. 53 MB/s | 2.3 MB 00:00 2023-10-06T01:38:46.591 INFO:teuthology.orchestra.run.smithi124.stdout:(80/120): pcre2-devel-10.32-2.el8.x86_64.rpm 23 MB/s | 605 kB 00:00 2023-10-06T01:38:46.601 INFO:teuthology.orchestra.run.smithi124.stdout:(81/120): fuse-2.9.7-12.el8.x86_64.rpm 6.1 MB/s | 83 kB 00:00 2023-10-06T01:38:46.604 INFO:teuthology.orchestra.run.smithi124.stdout:(82/120): elfutils-libelf-devel-0.182-3.el8.x86 4.4 MB/s | 59 kB 00:00 2023-10-06T01:38:46.615 INFO:teuthology.orchestra.run.smithi124.stdout:(83/120): libquadmath-8.4.1-1.el8.x86_64.rpm 12 MB/s | 169 kB 00:00 2023-10-06T01:38:46.616 INFO:teuthology.orchestra.run.smithi124.stdout:(84/120): libcom_err-devel-1.45.6-1.el8.x86_64. 3.1 MB/s | 38 kB 00:00 2023-10-06T01:38:46.629 INFO:teuthology.orchestra.run.smithi124.stdout:(85/120): libsepol-devel-2.9-2.el8.x86_64.rpm 6.1 MB/s | 86 kB 00:00 2023-10-06T01:38:46.630 INFO:teuthology.orchestra.run.smithi124.stdout:(86/120): zlib-devel-1.2.11-17.el8.x86_64.rpm 4.3 MB/s | 58 kB 00:00 2023-10-06T01:38:46.643 INFO:teuthology.orchestra.run.smithi124.stdout:(87/120): librabbitmq-0.9.0-3.el8.x86_64.rpm 3.6 MB/s | 47 kB 00:00 2023-10-06T01:38:46.649 INFO:teuthology.orchestra.run.smithi124.stdout:(88/120): krb5-devel-1.18.2-8.3.el8_4.x86_64.rp 29 MB/s | 559 kB 00:00 2023-10-06T01:38:46.662 INFO:teuthology.orchestra.run.smithi124.stdout:(89/120): libgfortran-8.4.1-1.el8.x86_64.rpm 33 MB/s | 639 kB 00:00 2023-10-06T01:38:46.663 INFO:teuthology.orchestra.run.smithi124.stdout:(90/120): xfsprogs-devel-5.0.0-8.el8.x86_64.rpm 4.9 MB/s | 70 kB 00:00 2023-10-06T01:38:46.676 INFO:teuthology.orchestra.run.smithi124.stdout:(91/120): libselinux-devel-2.9-5.el8.x86_64.rpm 14 MB/s | 200 kB 00:00 2023-10-06T01:38:46.678 INFO:teuthology.orchestra.run.smithi124.stdout:(92/120): libuuid-devel-2.32.1-27.el8.x86_64.rp 6.7 MB/s | 97 kB 00:00 2023-10-06T01:38:46.690 INFO:teuthology.orchestra.run.smithi124.stdout:(93/120): python3-prettytable-0.7.2-14.el8.noar 3.5 MB/s | 44 kB 00:00 2023-10-06T01:38:46.703 INFO:teuthology.orchestra.run.smithi124.stdout:(94/120): python3-jsonpatch-1.21-2.el8.noarch.r 2.1 MB/s | 27 kB 00:00 2023-10-06T01:38:46.717 INFO:teuthology.orchestra.run.smithi124.stdout:(95/120): jq-1.5-12.el8.x86_64.rpm 11 MB/s | 161 kB 00:00 2023-10-06T01:38:46.734 INFO:teuthology.orchestra.run.smithi124.stdout:(96/120): flex-2.6.1-9.el8.x86_64.rpm 19 MB/s | 320 kB 00:00 2023-10-06T01:38:46.752 INFO:teuthology.orchestra.run.smithi124.stdout:(97/120): python3-werkzeug-0.12.2-4.el8.noarch. 25 MB/s | 457 kB 00:00 2023-10-06T01:38:46.830 INFO:teuthology.orchestra.run.smithi124.stdout:(98/120): python3-babel-2.5.1-5.el8.noarch.rpm 62 MB/s | 4.8 MB 00:00 2023-10-06T01:38:46.842 INFO:teuthology.orchestra.run.smithi124.stdout:(99/120): python3-markupsafe-0.23-19.el8.x86_64 3.1 MB/s | 39 kB 00:00 2023-10-06T01:38:46.851 INFO:teuthology.orchestra.run.smithi027.stdout:(115/120): ceph-test-17.2.6-1296.g736ac5ba.el8. 12 MB/s | 52 MB 00:04 2023-10-06T01:38:46.884 INFO:teuthology.orchestra.run.smithi027.stdout:(116/120): openblas-0.3.12-1.el8.x86_64.rpm 8.7 MB/s | 4.6 MB 00:00 2023-10-06T01:38:46.900 INFO:teuthology.orchestra.run.smithi027.stdout:(117/120): openblas-threads-0.3.12-1.el8.x86_64 8.7 MB/s | 4.7 MB 00:00 2023-10-06T01:38:46.909 INFO:teuthology.orchestra.run.smithi027.stdout:(118/120): automake-1.16.1-7.el8.noarch.rpm 12 MB/s | 713 kB 00:00 2023-10-06T01:38:46.946 INFO:teuthology.orchestra.run.smithi124.stdout:(100/120): python3-scipy-1.0.0-20.module+el8.1. 54 MB/s | 15 MB 00:00 2023-10-06T01:38:46.954 INFO:teuthology.orchestra.run.smithi124.stdout:(101/120): bison-3.0.4-10.el8.x86_64.rpm 6.0 MB/s | 688 kB 00:00 2023-10-06T01:38:46.963 INFO:teuthology.orchestra.run.smithi124.stdout:(102/120): lttng-ust-2.8.1-11.el8.x86_64.rpm 15 MB/s | 259 kB 00:00 2023-10-06T01:38:46.972 INFO:teuthology.orchestra.run.smithi124.stdout:(103/120): python3-numpy-f2py-1.14.3-9.el8.x86_ 13 MB/s | 225 kB 00:00 2023-10-06T01:38:46.979 INFO:teuthology.orchestra.run.smithi124.stdout:(104/120): python3-pyasn1-0.3.7-6.el8.noarch.rp 8.3 MB/s | 126 kB 00:00 2023-10-06T01:38:46.993 INFO:teuthology.orchestra.run.smithi124.stdout:(105/120): oniguruma-6.8.2-2.el8.x86_64.rpm 12 MB/s | 187 kB 00:00 2023-10-06T01:38:47.014 INFO:teuthology.orchestra.run.smithi124.stdout:(106/120): python3-jinja2-2.10.1-2.el8_0.noarch 26 MB/s | 538 kB 00:00 2023-10-06T01:38:47.030 INFO:teuthology.orchestra.run.smithi124.stdout:(107/120): python3-numpy-1.14.3-9.el8.x86_64.rp 65 MB/s | 3.7 MB 00:00 2023-10-06T01:38:47.033 INFO:teuthology.orchestra.run.smithi124.stdout:(108/120): socat-1.7.3.3-2.el8.x86_64.rpm 15 MB/s | 302 kB 00:00 2023-10-06T01:38:47.044 INFO:teuthology.orchestra.run.smithi124.stdout:(109/120): python3-mako-1.0.6-13.el8.noarch.rpm 11 MB/s | 157 kB 00:00 2023-10-06T01:38:47.048 INFO:teuthology.orchestra.run.smithi124.stdout:(110/120): python3-pyasn1-modules-0.3.7-6.el8.n 7.4 MB/s | 110 kB 00:00 2023-10-06T01:38:47.061 INFO:teuthology.orchestra.run.smithi124.stdout:(111/120): librdkafka-0.11.4-1.el8.x86_64.rpm 21 MB/s | 353 kB 00:00 2023-10-06T01:38:47.062 INFO:teuthology.orchestra.run.smithi124.stdout:(112/120): python3-jsonpointer-1.10-11.el8.noar 1.4 MB/s | 20 kB 00:00 2023-10-06T01:38:47.075 INFO:teuthology.orchestra.run.smithi124.stdout:(113/120): python3-pytz-2017.2-9.el8.noarch.rpm 4.1 MB/s | 54 kB 00:00 2023-10-06T01:38:47.082 INFO:teuthology.orchestra.run.smithi124.stdout:(114/120): libtool-2.4.6-25.el8.x86_64.rpm 34 MB/s | 709 kB 00:00 2023-10-06T01:38:47.125 INFO:teuthology.orchestra.run.smithi027.stdout:(119/120): librados2-17.2.6-1296.g736ac5ba.el8. 14 MB/s | 3.5 MB 00:00 2023-10-06T01:38:47.148 INFO:teuthology.orchestra.run.smithi124.stdout:(115/120): openblas-threads-0.3.12-1.el8.x86_64 65 MB/s | 4.7 MB 00:00 2023-10-06T01:38:47.167 INFO:teuthology.orchestra.run.smithi124.stdout:(116/120): openblas-0.3.12-1.el8.x86_64.rpm 54 MB/s | 4.6 MB 00:00 2023-10-06T01:38:47.174 INFO:teuthology.orchestra.run.smithi124.stdout:(117/120): automake-1.16.1-7.el8.noarch.rpm 27 MB/s | 713 kB 00:00 2023-10-06T01:38:47.184 INFO:teuthology.orchestra.run.smithi124.stdout:(118/120): thrift-0.13.0-2.el8.x86_64.rpm 2.1 MB/s | 1.7 MB 00:00 2023-10-06T01:38:47.415 INFO:teuthology.orchestra.run.smithi124.stdout:(119/120): librados2-17.2.6-1296.g736ac5ba.el8. 14 MB/s | 3.5 MB 00:00 2023-10-06T01:38:48.581 INFO:teuthology.orchestra.run.smithi027.stdout:(120/120): librbd1-17.2.6-1296.g736ac5ba.el8.x8 2.1 MB/s | 3.6 MB 00:01 2023-10-06T01:38:48.585 INFO:teuthology.orchestra.run.smithi027.stdout:-------------------------------------------------------------------------------- 2023-10-06T01:38:48.586 INFO:teuthology.orchestra.run.smithi027.stdout:Total 17 MB/s | 201 MB 00:11 2023-10-06T01:38:48.645 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction check 2023-10-06T01:38:48.774 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction check succeeded. 2023-10-06T01:38:48.775 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction test 2023-10-06T01:38:48.979 INFO:teuthology.orchestra.run.smithi124.stdout:(120/120): librbd1-17.2.6-1296.g736ac5ba.el8.x8 2.0 MB/s | 3.6 MB 00:01 2023-10-06T01:38:48.984 INFO:teuthology.orchestra.run.smithi124.stdout:-------------------------------------------------------------------------------- 2023-10-06T01:38:48.984 INFO:teuthology.orchestra.run.smithi124.stdout:Total 14 MB/s | 201 MB 00:14 2023-10-06T01:38:49.032 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction check 2023-10-06T01:38:49.145 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction check succeeded. 2023-10-06T01:38:49.145 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction test 2023-10-06T01:38:50.501 INFO:teuthology.orchestra.run.smithi027.stdout:Transaction test succeeded. 2023-10-06T01:38:50.506 INFO:teuthology.orchestra.run.smithi027.stdout:Running transaction 2023-10-06T01:38:50.848 INFO:teuthology.orchestra.run.smithi124.stdout:Transaction test succeeded. 2023-10-06T01:38:50.852 INFO:teuthology.orchestra.run.smithi124.stdout:Running transaction 2023-10-06T01:38:55.534 INFO:teuthology.orchestra.run.smithi124.stdout: Preparing : 1/1 2023-10-06T01:38:55.594 INFO:teuthology.orchestra.run.smithi027.stdout: Preparing : 1/1 2023-10-06T01:38:55.795 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/122 2023-10-06T01:38:55.831 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/122 2023-10-06T01:38:55.979 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 2/122 2023-10-06T01:38:56.005 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 2/122 2023-10-06T01:38:56.125 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 2/122 2023-10-06T01:38:56.152 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 2/122 2023-10-06T01:38:56.304 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/122 2023-10-06T01:38:56.347 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/122 2023-10-06T01:38:56.711 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/122 2023-10-06T01:38:56.778 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/122 2023-10-06T01:38:56.938 INFO:teuthology.orchestra.run.smithi124.stdout: Upgrading : librados2-2:17.2.6-1296.g736ac5ba.el8.x86_64 4/122 2023-10-06T01:38:56.989 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librados2-2:17.2.6-1296.g736ac5ba.el8.x86_64 4/122 2023-10-06T01:38:57.337 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: librados2-2:17.2.6-1296.g736ac5ba.el8.x86_64 4/122 2023-10-06T01:38:57.406 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-2:17.2.6-1296.g736ac5ba.el8.x86_64 4/122 2023-10-06T01:38:57.546 INFO:teuthology.orchestra.run.smithi124.stdout: Upgrading : librbd1-2:17.2.6-1296.g736ac5ba.el8.x86_64 5/122 2023-10-06T01:38:57.614 INFO:teuthology.orchestra.run.smithi027.stdout: Upgrading : librbd1-2:17.2.6-1296.g736ac5ba.el8.x86_64 5/122 2023-10-06T01:38:57.676 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: librbd1-2:17.2.6-1296.g736ac5ba.el8.x86_64 5/122 2023-10-06T01:38:57.745 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-2:17.2.6-1296.g736ac5ba.el8.x86_64 5/122 2023-10-06T01:38:57.896 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libcephfs2-2:17.2.6-1296.g736ac5ba.el8.x86_64 6/122 2023-10-06T01:38:57.995 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: libcephfs2-2:17.2.6-1296.g736ac5ba.el8.x86_64 6/122 2023-10-06T01:38:58.184 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-rados-2:17.2.6-1296.g736ac5ba.el8.x86_64 7/122 2023-10-06T01:38:58.330 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs2-2:17.2.6-1296.g736ac5ba.el8.x86_64 6/122 2023-10-06T01:38:58.440 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephfs2-2:17.2.6-1296.g736ac5ba.el8.x86_64 6/122 2023-10-06T01:38:58.492 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/122 2023-10-06T01:38:58.661 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rados-2:17.2.6-1296.g736ac5ba.el8.x86_64 7/122 2023-10-06T01:38:59.002 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/122 2023-10-06T01:38:59.199 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/122 2023-10-06T01:38:59.387 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : librgw2-2:17.2.6-1296.g736ac5ba.el8.x86_64 10/122 2023-10-06T01:38:59.504 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: librgw2-2:17.2.6-1296.g736ac5ba.el8.x86_64 10/122 2023-10-06T01:38:59.699 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/122 2023-10-06T01:38:59.741 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/122 2023-10-06T01:38:59.838 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librgw2-2:17.2.6-1296.g736ac5ba.el8.x86_64 10/122 2023-10-06T01:38:59.987 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librgw2-2:17.2.6-1296.g736ac5ba.el8.x86_64 10/122 2023-10-06T01:39:00.218 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/122 2023-10-06T01:39:00.362 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/122 2023-10-06T01:39:00.388 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/122 2023-10-06T01:39:00.570 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libcephsqlite-2:17.2.6-1296.g736ac5ba.el8.x86_64 13/122 2023-10-06T01:39:00.571 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephsqlite-2:17.2.6-1296.g736ac5ba.el8.x86_64 13/122 2023-10-06T01:39:00.679 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: libcephsqlite-2:17.2.6-1296.g736ac5ba.el8.x86_64 13/122 2023-10-06T01:39:00.713 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libcephsqlite-2:17.2.6-1296.g736ac5ba.el8.x86_64 13/122 2023-10-06T01:39:00.887 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libradosstriper1-2:17.2.6-1296.g736ac5ba.el8.x86 14/122 2023-10-06T01:39:00.889 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libradosstriper1-2:17.2.6-1296.g736ac5ba.el8.x86 14/122 2023-10-06T01:39:00.965 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libradosstriper1-2:17.2.6-1296.g736ac5ba.el8.x86 14/122 2023-10-06T01:39:00.967 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: libradosstriper1-2:17.2.6-1296.g736ac5ba.el8.x86 14/122 2023-10-06T01:39:01.126 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 15/122 2023-10-06T01:39:01.161 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 15/122 2023-10-06T01:39:01.367 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 16/122 2023-10-06T01:39:01.429 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 16/122 2023-10-06T01:39:01.553 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 17/122 2023-10-06T01:39:01.669 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 17/122 2023-10-06T01:39:01.693 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 18/122 2023-10-06T01:39:01.819 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 18/122 2023-10-06T01:39:01.830 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 19/122 2023-10-06T01:39:01.945 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 19/122 2023-10-06T01:39:01.970 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 19/122 2023-10-06T01:39:02.081 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 19/122 2023-10-06T01:39:02.138 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 20/122 2023-10-06T01:39:02.245 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 20/122 2023-10-06T01:39:02.723 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 20/122 2023-10-06T01:39:02.800 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 20/122 2023-10-06T01:39:02.871 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 21/122 2023-10-06T01:39:02.970 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 21/122 2023-10-06T01:39:03.421 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 21/122 2023-10-06T01:39:03.511 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 21/122 2023-10-06T01:39:03.712 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : openblas-0.3.12-1.el8.x86_64 22/122 2023-10-06T01:39:03.888 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openblas-0.3.12-1.el8.x86_64 22/122 2023-10-06T01:39:04.459 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 22/122 2023-10-06T01:39:04.657 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 22/122 2023-10-06T01:39:04.731 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 23/122 2023-10-06T01:39:04.892 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 23/122 2023-10-06T01:39:04.972 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 24/122 2023-10-06T01:39:05.134 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 24/122 2023-10-06T01:39:05.223 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/122 2023-10-06T01:39:05.378 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/122 2023-10-06T01:39:05.426 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-ceph-common-2:17.2.6-1296.g736ac5ba.el8. 26/122 2023-10-06T01:39:05.545 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-common-2:17.2.6-1296.g736ac5ba.el8. 26/122 2023-10-06T01:39:05.700 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-ceph-argparse-2:17.2.6-1296.g736ac5ba.el 27/122 2023-10-06T01:39:05.765 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-ceph-argparse-2:17.2.6-1296.g736ac5ba.el 27/122 2023-10-06T01:39:05.959 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cephfs-2:17.2.6-1296.g736ac5ba.el8.x86_6 28/122 2023-10-06T01:39:06.050 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-cephfs-2:17.2.6-1296.g736ac5ba.el8.x86_6 28/122 2023-10-06T01:39:06.187 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 29/122 2023-10-06T01:39:06.294 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 29/122 2023-10-06T01:39:06.455 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 30/122 2023-10-06T01:39:06.581 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 30/122 2023-10-06T01:39:06.662 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 31/122 2023-10-06T01:39:06.844 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 31/122 2023-10-06T01:39:08.877 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 31/122 2023-10-06T01:39:08.922 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 31/122 2023-10-06T01:39:09.232 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 32/122 2023-10-06T01:39:09.268 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 32/122 2023-10-06T01:39:10.256 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/122 2023-10-06T01:39:10.284 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/122 2023-10-06T01:39:10.565 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 34/122 2023-10-06T01:39:10.618 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 34/122 2023-10-06T01:39:10.851 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 35/122 2023-10-06T01:39:11.065 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rsa-4.9-2.el8.noarch 36/122 2023-10-06T01:39:11.167 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 35/122 2023-10-06T01:39:11.249 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 37/122 2023-10-06T01:39:11.402 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-rsa-4.9-2.el8.noarch 36/122 2023-10-06T01:39:11.474 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rgw-2:17.2.6-1296.g736ac5ba.el8.x86_64 38/122 2023-10-06T01:39:11.604 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 37/122 2023-10-06T01:39:11.665 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-rbd-2:17.2.6-1296.g736ac5ba.el8.x86_64 39/122 2023-10-06T01:39:11.844 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-rgw-2:17.2.6-1296.g736ac5ba.el8.x86_64 38/122 2023-10-06T01:39:11.936 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : librados-devel-2:17.2.6-1296.g736ac5ba.el8.x86_6 40/122 2023-10-06T01:39:12.034 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-rbd-2:17.2.6-1296.g736ac5ba.el8.x86_64 39/122 2023-10-06T01:39:12.179 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : automake-1.16.1-7.el8.noarch 41/122 2023-10-06T01:39:12.309 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : librados-devel-2:17.2.6-1296.g736ac5ba.el8.x86_6 40/122 2023-10-06T01:39:12.361 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 42/122 2023-10-06T01:39:12.534 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : automake-1.16.1-7.el8.noarch 41/122 2023-10-06T01:39:12.545 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 43/122 2023-10-06T01:39:12.710 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 42/122 2023-10-06T01:39:12.741 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 44/122 2023-10-06T01:39:12.895 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 45/122 2023-10-06T01:39:12.916 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 43/122 2023-10-06T01:39:12.963 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 45/122 2023-10-06T01:39:13.097 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 44/122 2023-10-06T01:39:13.121 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : jq-1.5-12.el8.x86_64 46/122 2023-10-06T01:39:13.282 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 47/122 2023-10-06T01:39:13.286 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 45/122 2023-10-06T01:39:13.361 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 45/122 2023-10-06T01:39:13.455 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libuuid-devel-2.32.1-27.el8.x86_64 48/122 2023-10-06T01:39:13.528 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : jq-1.5-12.el8.x86_64 46/122 2023-10-06T01:39:13.625 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libsepol-devel-2.9-2.el8.x86_64 49/122 2023-10-06T01:39:13.689 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 47/122 2023-10-06T01:39:13.815 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcom_err-devel-1.45.6-1.el8.x86_64 50/122 2023-10-06T01:39:13.844 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libuuid-devel-2.32.1-27.el8.x86_64 48/122 2023-10-06T01:39:13.981 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : fuse-2.9.7-12.el8.x86_64 51/122 2023-10-06T01:39:14.033 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libsepol-devel-2.9-2.el8.x86_64 49/122 2023-10-06T01:39:14.196 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libcom_err-devel-1.45.6-1.el8.x86_64 50/122 2023-10-06T01:39:14.275 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 52/122 2023-10-06T01:39:14.355 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : fuse-2.9.7-12.el8.x86_64 51/122 2023-10-06T01:39:14.462 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 53/122 2023-10-06T01:39:14.490 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 52/122 2023-10-06T01:39:14.624 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 54/122 2023-10-06T01:39:14.676 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 53/122 2023-10-06T01:39:14.882 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 54/122 2023-10-06T01:39:14.931 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/122 2023-10-06T01:39:15.170 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/122 2023-10-06T01:39:15.176 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/122 2023-10-06T01:39:15.407 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/122 2023-10-06T01:39:15.423 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/122 2023-10-06T01:39:15.680 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : krb5-devel-1.18.2-8.3.el8_4.x86_64 58/122 2023-10-06T01:39:15.693 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/122 2023-10-06T01:39:15.846 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 59/122 2023-10-06T01:39:15.987 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : krb5-devel-1.18.2-8.3.el8_4.x86_64 58/122 2023-10-06T01:39:16.027 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 60/122 2023-10-06T01:39:16.178 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 59/122 2023-10-06T01:39:16.275 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 61/122 2023-10-06T01:39:16.358 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 60/122 2023-10-06T01:39:16.504 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 62/122 2023-10-06T01:39:16.625 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 61/122 2023-10-06T01:39:16.702 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 63/122 2023-10-06T01:39:16.877 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 62/122 2023-10-06T01:39:16.878 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 64/122 2023-10-06T01:39:17.057 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 65/122 2023-10-06T01:39:17.075 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 63/122 2023-10-06T01:39:17.242 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 66/122 2023-10-06T01:39:17.253 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 64/122 2023-10-06T01:39:17.429 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 65/122 2023-10-06T01:39:17.443 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 67/122 2023-10-06T01:39:17.662 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 66/122 2023-10-06T01:39:17.674 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 68/122 2023-10-06T01:39:17.889 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 67/122 2023-10-06T01:39:17.906 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 69/122 2023-10-06T01:39:18.085 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 70/122 2023-10-06T01:39:18.112 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 68/122 2023-10-06T01:39:18.313 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 69/122 2023-10-06T01:39:18.318 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 71/122 2023-10-06T01:39:18.499 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 70/122 2023-10-06T01:39:18.669 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 72/122 2023-10-06T01:39:18.748 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 71/122 2023-10-06T01:39:18.869 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 73/122 2023-10-06T01:39:18.964 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 72/122 2023-10-06T01:39:19.177 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-portend-2.6-1.el8.noarch 74/122 2023-10-06T01:39:19.191 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 73/122 2023-10-06T01:39:19.479 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-portend-2.6-1.el8.noarch 74/122 2023-10-06T01:39:19.726 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 75/122 2023-10-06T01:39:19.780 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 75/122 2023-10-06T01:39:19.944 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 76/122 2023-10-06T01:39:20.016 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 76/122 2023-10-06T01:39:20.187 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 77/122 2023-10-06T01:39:20.297 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 77/122 2023-10-06T01:39:21.334 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 78/122 2023-10-06T01:39:21.534 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 78/122 2023-10-06T01:39:21.699 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 79/122 2023-10-06T01:39:21.845 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 79/122 2023-10-06T01:39:21.921 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 80/122 2023-10-06T01:39:22.114 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 80/122 2023-10-06T01:39:22.283 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 81/122 2023-10-06T01:39:22.441 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 81/122 2023-10-06T01:39:22.526 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 82/122 2023-10-06T01:39:22.721 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 82/122 2023-10-06T01:39:22.840 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 83/122 2023-10-06T01:39:23.064 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 84/122 2023-10-06T01:39:23.078 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 83/122 2023-10-06T01:39:23.268 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 85/122 2023-10-06T01:39:23.312 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 84/122 2023-10-06T01:39:23.510 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 86/122 2023-10-06T01:39:23.538 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 85/122 2023-10-06T01:39:23.736 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 86/122 2023-10-06T01:39:28.505 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-common-2:17.2.6-1296.g736ac5ba.el8.x86_64 87/122 2023-10-06T01:39:28.742 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-common-2:17.2.6-1296.g736ac5ba.el8.x86_64 87/122 2023-10-06T01:39:29.173 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:17.2.6-1296.g736ac5ba.el8.x86_64 87/122 2023-10-06T01:39:29.333 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-common-2:17.2.6-1296.g736ac5ba.el8.x86_64 87/122 2023-10-06T01:39:29.377 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-common-2:17.2.6-1296.g736ac5ba.el8.x86_64 87/122 2023-10-06T01:39:29.617 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-base-2:17.2.6-1296.g736ac5ba.el8.x86_64 88/122 2023-10-06T01:39:29.723 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-base-2:17.2.6-1296.g736ac5ba.el8.x86_64 88/122 2023-10-06T01:39:29.867 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-selinux-2:17.2.6-1296.g736ac5ba.el8.x86_64 89/122 2023-10-06T01:39:30.005 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-common-2:17.2.6-1296.g736ac5ba.el8.x86_64 87/122 2023-10-06T01:39:30.914 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-base-2:17.2.6-1296.g736ac5ba.el8.x86_64 88/122 2023-10-06T01:39:31.002 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-base-2:17.2.6-1296.g736ac5ba.el8.x86_64 88/122 2023-10-06T01:39:31.126 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-selinux-2:17.2.6-1296.g736ac5ba.el8.x86_64 89/122 2023-10-06T01:39:47.384 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-selinux-2:17.2.6-1296.g736ac5ba.el8.x86_64 89/122 2023-10-06T01:39:47.384 INFO:teuthology.orchestra.run.smithi124.stdout:skipping the directory /sys 2023-10-06T01:39:47.384 INFO:teuthology.orchestra.run.smithi124.stdout:skipping the directory /proc 2023-10-06T01:39:47.384 INFO:teuthology.orchestra.run.smithi124.stdout:skipping the directory /dev 2023-10-06T01:39:47.384 INFO:teuthology.orchestra.run.smithi124.stdout:skipping the directory /run 2023-10-06T01:39:47.384 INFO:teuthology.orchestra.run.smithi124.stdout:skipping the directory /mnt 2023-10-06T01:39:47.384 INFO:teuthology.orchestra.run.smithi124.stdout:skipping the directory /var/tmp 2023-10-06T01:39:47.385 INFO:teuthology.orchestra.run.smithi124.stdout:skipping the directory /home 2023-10-06T01:39:47.385 INFO:teuthology.orchestra.run.smithi124.stdout:skipping the directory /tmp 2023-10-06T01:39:47.385 INFO:teuthology.orchestra.run.smithi124.stdout:skipping the directory /dev 2023-10-06T01:39:47.385 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-10-06T01:39:47.749 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-volume-2:17.2.6-1296.g736ac5ba.el8.noarch 90/122 2023-10-06T01:39:49.235 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-volume-2:17.2.6-1296.g736ac5ba.el8.noarch 90/122 2023-10-06T01:39:49.398 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-osd-2:17.2.6-1296.g736ac5ba.el8.x86_64 91/122 2023-10-06T01:39:49.568 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-selinux-2:17.2.6-1296.g736ac5ba.el8.x86_64 89/122 2023-10-06T01:39:49.568 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /sys 2023-10-06T01:39:49.568 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /proc 2023-10-06T01:39:49.568 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /dev 2023-10-06T01:39:49.568 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /run 2023-10-06T01:39:49.568 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /mnt 2023-10-06T01:39:49.569 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /var/tmp 2023-10-06T01:39:49.569 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /home 2023-10-06T01:39:49.569 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /tmp 2023-10-06T01:39:49.569 INFO:teuthology.orchestra.run.smithi027.stdout:skipping the directory /dev 2023-10-06T01:39:49.569 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:39:49.620 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-osd-2:17.2.6-1296.g736ac5ba.el8.x86_64 91/122 2023-10-06T01:39:49.765 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-mds-2:17.2.6-1296.g736ac5ba.el8.x86_64 92/122 2023-10-06T01:39:49.974 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-volume-2:17.2.6-1296.g736ac5ba.el8.noarch 90/122 2023-10-06T01:39:50.163 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-mds-2:17.2.6-1296.g736ac5ba.el8.x86_64 92/122 2023-10-06T01:39:50.306 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-mon-2:17.2.6-1296.g736ac5ba.el8.x86_64 93/122 2023-10-06T01:39:50.323 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-mon-2:17.2.6-1296.g736ac5ba.el8.x86_64 93/122 2023-10-06T01:39:51.550 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-volume-2:17.2.6-1296.g736ac5ba.el8.noarch 90/122 2023-10-06T01:39:51.703 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-osd-2:17.2.6-1296.g736ac5ba.el8.x86_64 91/122 2023-10-06T01:39:51.939 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-osd-2:17.2.6-1296.g736ac5ba.el8.x86_64 91/122 2023-10-06T01:39:52.091 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mds-2:17.2.6-1296.g736ac5ba.el8.x86_64 92/122 2023-10-06T01:39:52.512 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mds-2:17.2.6-1296.g736ac5ba.el8.x86_64 92/122 2023-10-06T01:39:52.674 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mon-2:17.2.6-1296.g736ac5ba.el8.x86_64 93/122 2023-10-06T01:39:52.692 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mon-2:17.2.6-1296.g736ac5ba.el8.x86_64 93/122 2023-10-06T01:39:53.175 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: cephadm-2:17.2.6-1296.g736ac5ba.el8.noarch 94/122 2023-10-06T01:39:53.309 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : cephadm-2:17.2.6-1296.g736ac5ba.el8.noarch 94/122 2023-10-06T01:39:53.504 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-1296.g736ac5ba.e 95/122 2023-10-06T01:39:53.718 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-1296.g736ac5ba. 96/122 2023-10-06T01:39:53.932 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-1296.g736ac5ba.el8.noa 97/122 2023-10-06T01:39:54.150 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-1296.g736ac5ba.el8.noa 97/122 2023-10-06T01:39:54.364 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-1296.g736ac5ba.el8.n 98/122 2023-10-06T01:39:55.335 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-1296.g736ac5ba.el8.n 98/122 2023-10-06T01:39:55.505 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-1296.g736 99/122 2023-10-06T01:39:55.604 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-1296.g736 99/122 2023-10-06T01:39:55.785 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cephadm-2:17.2.6-1296.g736ac5ba.el8.noarch 94/122 2023-10-06T01:39:55.914 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephadm-2:17.2.6-1296.g736ac5ba.el8.noarch 94/122 2023-10-06T01:39:55.978 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-1296.g736ac5ba.el 100/122 2023-10-06T01:39:56.105 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-1296.g736ac5ba.e 95/122 2023-10-06T01:39:56.141 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-mgr-2:17.2.6-1296.g736ac5ba.el8.x86_64 101/122 2023-10-06T01:39:56.176 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-mgr-2:17.2.6-1296.g736ac5ba.el8.x86_64 101/122 2023-10-06T01:39:56.311 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-1296.g736ac5ba. 96/122 2023-10-06T01:39:56.321 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-mgr-rook-2:17.2.6-1296.g736ac5ba.el8.noarch 102/122 2023-10-06T01:39:56.333 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-1296.g736ac5ba.el8.noarch 102/122 2023-10-06T01:39:56.484 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-1296.g736ac5ba.el8.noa 97/122 2023-10-06T01:39:56.712 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-1296.g736ac5ba.el8.noa 97/122 2023-10-06T01:39:56.927 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-1296.g736ac5ba.el8.n 98/122 2023-10-06T01:39:57.921 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-2:17.2.6-1296.g736ac5ba.el8.x86_64 103/122 2023-10-06T01:39:57.959 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-1296.g736ac5ba.el8.n 98/122 2023-10-06T01:39:58.114 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-1296.g736 99/122 2023-10-06T01:39:58.147 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-radosgw-2:17.2.6-1296.g736ac5ba.el8.x86_64 104/122 2023-10-06T01:39:58.218 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-1296.g736 99/122 2023-10-06T01:39:58.225 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-1296.g736ac5ba.el8.x86_64 104/122 2023-10-06T01:39:58.372 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-1296.g736ac 105/122 2023-10-06T01:39:58.415 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-1296.g736ac 105/122 2023-10-06T01:39:58.563 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : cephfs-mirror-2:17.2.6-1296.g736ac5ba.el8.x86_64 106/122 2023-10-06T01:39:58.617 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-1296.g736ac5ba.el 100/122 2023-10-06T01:39:58.777 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-2:17.2.6-1296.g736ac5ba.el8.x86_64 101/122 2023-10-06T01:39:58.816 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-2:17.2.6-1296.g736ac5ba.el8.x86_64 101/122 2023-10-06T01:39:58.911 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: cephfs-mirror-2:17.2.6-1296.g736ac5ba.el8.x86_64 106/122 2023-10-06T01:39:59.014 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-mgr-rook-2:17.2.6-1296.g736ac5ba.el8.noarch 102/122 2023-10-06T01:39:59.026 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-1296.g736ac5ba.el8.noarch 102/122 2023-10-06T01:39:59.105 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : rbd-mirror-2:17.2.6-1296.g736ac5ba.el8.x86_64 107/122 2023-10-06T01:40:00.719 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-2:17.2.6-1296.g736ac5ba.el8.x86_64 103/122 2023-10-06T01:40:00.910 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-radosgw-2:17.2.6-1296.g736ac5ba.el8.x86_64 104/122 2023-10-06T01:40:01.008 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-1296.g736ac5ba.el8.x86_64 104/122 2023-10-06T01:40:01.181 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-1296.g736ac 105/122 2023-10-06T01:40:01.227 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-1296.g736ac 105/122 2023-10-06T01:40:01.356 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephfs-mirror-2:17.2.6-1296.g736ac5ba.el8.x86_64 106/122 2023-10-06T01:40:01.726 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: cephfs-mirror-2:17.2.6-1296.g736ac5ba.el8.x86_64 106/122 2023-10-06T01:40:01.914 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-mirror-2:17.2.6-1296.g736ac5ba.el8.x86_64 107/122 2023-10-06T01:40:04.011 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: rbd-mirror-2:17.2.6-1296.g736ac5ba.el8.x86_64 107/122 2023-10-06T01:40:04.945 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-test-2:17.2.6-1296.g736ac5ba.el8.x86_64 108/122 2023-10-06T01:40:05.344 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : openssl-devel-1:1.1.1g-15.el8_3.x86_64 109/122 2023-10-06T01:40:05.529 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : ceph-fuse-2:17.2.6-1296.g736ac5ba.el8.x86_64 110/122 2023-10-06T01:40:05.744 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : xfsprogs-devel-5.0.0-8.el8.x86_64 111/122 2023-10-06T01:40:05.945 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libtool-2.4.6-25.el8.x86_64 112/122 2023-10-06T01:40:05.957 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 112/122 2023-10-06T01:40:06.125 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libcephfs-devel-2:17.2.6-1296.g736ac5ba.el8.x86_ 113/122 2023-10-06T01:40:06.283 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : elfutils-libelf-devel-0.182-3.el8.x86_64 114/122 2023-10-06T01:40:06.445 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : cephfs-top-2:17.2.6-1296.g736ac5ba.el8.noarch 115/122 2023-10-06T01:40:06.662 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : rbd-fuse-2:17.2.6-1296.g736ac5ba.el8.x86_64 116/122 2023-10-06T01:40:06.910 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : rbd-nbd-2:17.2.6-1296.g736ac5ba.el8.x86_64 117/122 2023-10-06T01:40:07.120 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : bison-3.0.4-10.el8.x86_64 118/122 2023-10-06T01:40:07.146 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: rbd-mirror-2:17.2.6-1296.g736ac5ba.el8.x86_64 107/122 2023-10-06T01:40:07.181 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 118/122 2023-10-06T01:40:07.404 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : flex-2.6.1-9.el8.x86_64 119/122 2023-10-06T01:40:07.412 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 119/122 2023-10-06T01:40:07.531 INFO:teuthology.orchestra.run.smithi124.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 120/122 2023-10-06T01:40:07.531 INFO:teuthology.orchestra.run.smithi124.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-10-06T01:40:07.740 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-10-06T01:40:07.740 INFO:teuthology.orchestra.run.smithi124.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 122/122 2023-10-06T01:40:08.136 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-test-2:17.2.6-1296.g736ac5ba.el8.x86_64 108/122 2023-10-06T01:40:08.512 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : openssl-devel-1:1.1.1g-15.el8_3.x86_64 109/122 2023-10-06T01:40:08.593 INFO:teuthology.orchestra.run.smithi124.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 122/122 2023-10-06T01:40:08.593 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-2:17.2.6-1296.g736ac5ba.el8.x86_64 1/122 2023-10-06T01:40:08.593 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-base-2:17.2.6-1296.g736ac5ba.el8.x86_64 2/122 2023-10-06T01:40:08.593 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-common-2:17.2.6-1296.g736ac5ba.el8.x86_64 3/122 2023-10-06T01:40:08.594 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-fuse-2:17.2.6-1296.g736ac5ba.el8.x86_64 4/122 2023-10-06T01:40:08.594 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-1296.g736ac 5/122 2023-10-06T01:40:08.594 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-mds-2:17.2.6-1296.g736ac5ba.el8.x86_64 6/122 2023-10-06T01:40:08.594 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-mgr-2:17.2.6-1296.g736ac5ba.el8.x86_64 7/122 2023-10-06T01:40:08.594 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-mon-2:17.2.6-1296.g736ac5ba.el8.x86_64 8/122 2023-10-06T01:40:08.594 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-osd-2:17.2.6-1296.g736ac5ba.el8.x86_64 9/122 2023-10-06T01:40:08.595 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-radosgw-2:17.2.6-1296.g736ac5ba.el8.x86_64 10/122 2023-10-06T01:40:08.595 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-selinux-2:17.2.6-1296.g736ac5ba.el8.x86_64 11/122 2023-10-06T01:40:08.595 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-test-2:17.2.6-1296.g736ac5ba.el8.x86_64 12/122 2023-10-06T01:40:08.595 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : cephfs-mirror-2:17.2.6-1296.g736ac5ba.el8.x86_64 13/122 2023-10-06T01:40:08.595 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libcephfs-devel-2:17.2.6-1296.g736ac5ba.el8.x86_ 14/122 2023-10-06T01:40:08.595 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libcephfs2-2:17.2.6-1296.g736ac5ba.el8.x86_64 15/122 2023-10-06T01:40:08.595 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libcephsqlite-2:17.2.6-1296.g736ac5ba.el8.x86_64 16/122 2023-10-06T01:40:08.595 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : librados-devel-2:17.2.6-1296.g736ac5ba.el8.x86_6 17/122 2023-10-06T01:40:08.596 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libradosstriper1-2:17.2.6-1296.g736ac5ba.el8.x86 18/122 2023-10-06T01:40:08.596 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : librgw2-2:17.2.6-1296.g736ac5ba.el8.x86_64 19/122 2023-10-06T01:40:08.596 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-ceph-argparse-2:17.2.6-1296.g736ac5ba.el 20/122 2023-10-06T01:40:08.596 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-ceph-common-2:17.2.6-1296.g736ac5ba.el8. 21/122 2023-10-06T01:40:08.596 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-cephfs-2:17.2.6-1296.g736ac5ba.el8.x86_6 22/122 2023-10-06T01:40:08.596 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-rados-2:17.2.6-1296.g736ac5ba.el8.x86_64 23/122 2023-10-06T01:40:08.596 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-rbd-2:17.2.6-1296.g736ac5ba.el8.x86_64 24/122 2023-10-06T01:40:08.596 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-rgw-2:17.2.6-1296.g736ac5ba.el8.x86_64 25/122 2023-10-06T01:40:08.597 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : rbd-fuse-2:17.2.6-1296.g736ac5ba.el8.x86_64 26/122 2023-10-06T01:40:08.597 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : rbd-mirror-2:17.2.6-1296.g736ac5ba.el8.x86_64 27/122 2023-10-06T01:40:08.597 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : rbd-nbd-2:17.2.6-1296.g736ac5ba.el8.x86_64 28/122 2023-10-06T01:40:08.597 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-1296.g736ac5ba. 29/122 2023-10-06T01:40:08.597 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-1296.g736ac5ba.el8.noa 30/122 2023-10-06T01:40:08.597 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-1296.g736ac5ba.el8.n 31/122 2023-10-06T01:40:08.597 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-1296.g736 32/122 2023-10-06T01:40:08.597 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-1296.g736ac5ba.el 33/122 2023-10-06T01:40:08.598 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-mgr-rook-2:17.2.6-1296.g736ac5ba.el8.noarch 34/122 2023-10-06T01:40:08.598 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-1296.g736ac5ba.e 35/122 2023-10-06T01:40:08.598 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : ceph-volume-2:17.2.6-1296.g736ac5ba.el8.noarch 36/122 2023-10-06T01:40:08.598 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : cephadm-2:17.2.6-1296.g736ac5ba.el8.noarch 37/122 2023-10-06T01:40:08.598 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : cephfs-top-2:17.2.6-1296.g736ac5ba.el8.noarch 38/122 2023-10-06T01:40:08.598 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/122 2023-10-06T01:40:08.598 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/122 2023-10-06T01:40:08.598 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/122 2023-10-06T01:40:08.599 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/122 2023-10-06T01:40:08.599 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/122 2023-10-06T01:40:08.599 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/122 2023-10-06T01:40:08.599 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/122 2023-10-06T01:40:08.599 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/122 2023-10-06T01:40:08.599 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/122 2023-10-06T01:40:08.599 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/122 2023-10-06T01:40:08.599 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/122 2023-10-06T01:40:08.600 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/122 2023-10-06T01:40:08.600 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/122 2023-10-06T01:40:08.600 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/122 2023-10-06T01:40:08.600 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/122 2023-10-06T01:40:08.600 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/122 2023-10-06T01:40:08.600 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/122 2023-10-06T01:40:08.600 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/122 2023-10-06T01:40:08.600 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/122 2023-10-06T01:40:08.601 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/122 2023-10-06T01:40:08.601 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/122 2023-10-06T01:40:08.601 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/122 2023-10-06T01:40:08.601 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/122 2023-10-06T01:40:08.601 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/122 2023-10-06T01:40:08.601 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/122 2023-10-06T01:40:08.601 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/122 2023-10-06T01:40:08.601 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/122 2023-10-06T01:40:08.602 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/122 2023-10-06T01:40:08.602 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/122 2023-10-06T01:40:08.602 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/122 2023-10-06T01:40:08.602 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/122 2023-10-06T01:40:08.602 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/122 2023-10-06T01:40:08.602 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/122 2023-10-06T01:40:08.602 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 72/122 2023-10-06T01:40:08.602 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 73/122 2023-10-06T01:40:08.603 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 74/122 2023-10-06T01:40:08.603 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/122 2023-10-06T01:40:08.603 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 76/122 2023-10-06T01:40:08.603 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 77/122 2023-10-06T01:40:08.603 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/122 2023-10-06T01:40:08.603 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : openssl-devel-1:1.1.1g-15.el8_3.x86_64 79/122 2023-10-06T01:40:08.603 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 80/122 2023-10-06T01:40:08.604 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 81/122 2023-10-06T01:40:08.604 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 82/122 2023-10-06T01:40:08.604 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : elfutils-libelf-devel-0.182-3.el8.x86_64 83/122 2023-10-06T01:40:08.604 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 84/122 2023-10-06T01:40:08.604 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libcom_err-devel-1.45.6-1.el8.x86_64 85/122 2023-10-06T01:40:08.604 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libsepol-devel-2.9-2.el8.x86_64 86/122 2023-10-06T01:40:08.604 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 87/122 2023-10-06T01:40:08.604 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 88/122 2023-10-06T01:40:08.605 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : krb5-devel-1.18.2-8.3.el8_4.x86_64 89/122 2023-10-06T01:40:08.605 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 90/122 2023-10-06T01:40:08.605 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : xfsprogs-devel-5.0.0-8.el8.x86_64 91/122 2023-10-06T01:40:08.605 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 92/122 2023-10-06T01:40:08.605 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libuuid-devel-2.32.1-27.el8.x86_64 93/122 2023-10-06T01:40:08.605 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 94/122 2023-10-06T01:40:08.605 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 95/122 2023-10-06T01:40:08.606 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 96/122 2023-10-06T01:40:08.606 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : jq-1.5-12.el8.x86_64 97/122 2023-10-06T01:40:08.606 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : flex-2.6.1-9.el8.x86_64 98/122 2023-10-06T01:40:08.606 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 99/122 2023-10-06T01:40:08.606 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 100/122 2023-10-06T01:40:08.606 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 101/122 2023-10-06T01:40:08.607 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : bison-3.0.4-10.el8.x86_64 102/122 2023-10-06T01:40:08.607 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 103/122 2023-10-06T01:40:08.607 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 104/122 2023-10-06T01:40:08.607 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 105/122 2023-10-06T01:40:08.607 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 106/122 2023-10-06T01:40:08.607 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/122 2023-10-06T01:40:08.607 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 108/122 2023-10-06T01:40:08.608 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 109/122 2023-10-06T01:40:08.608 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 110/122 2023-10-06T01:40:08.608 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 111/122 2023-10-06T01:40:08.608 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 112/122 2023-10-06T01:40:08.608 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 113/122 2023-10-06T01:40:08.608 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 114/122 2023-10-06T01:40:08.608 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 115/122 2023-10-06T01:40:08.609 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 116/122 2023-10-06T01:40:08.609 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 117/122 2023-10-06T01:40:08.609 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : automake-1.16.1-7.el8.noarch 118/122 2023-10-06T01:40:08.609 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : librados2-2:17.2.6-1296.g736ac5ba.el8.x86_64 119/122 2023-10-06T01:40:08.609 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 120/122 2023-10-06T01:40:08.609 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : librbd1-2:17.2.6-1296.g736ac5ba.el8.x86_64 121/122 2023-10-06T01:40:08.699 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : ceph-fuse-2:17.2.6-1296.g736ac5ba.el8.x86_64 110/122 2023-10-06T01:40:08.961 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : xfsprogs-devel-5.0.0-8.el8.x86_64 111/122 2023-10-06T01:40:09.113 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libtool-2.4.6-25.el8.x86_64 112/122 2023-10-06T01:40:09.124 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 112/122 2023-10-06T01:40:09.301 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libcephfs-devel-2:17.2.6-1296.g736ac5ba.el8.x86_ 113/122 2023-10-06T01:40:09.451 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : elfutils-libelf-devel-0.182-3.el8.x86_64 114/122 2023-10-06T01:40:09.647 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : cephfs-top-2:17.2.6-1296.g736ac5ba.el8.noarch 115/122 2023-10-06T01:40:09.847 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-fuse-2:17.2.6-1296.g736ac5ba.el8.x86_64 116/122 2023-10-06T01:40:10.094 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : rbd-nbd-2:17.2.6-1296.g736ac5ba.el8.x86_64 117/122 2023-10-06T01:40:10.296 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : bison-3.0.4-10.el8.x86_64 118/122 2023-10-06T01:40:10.357 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 118/122 2023-10-06T01:40:10.538 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : flex-2.6.1-9.el8.x86_64 119/122 2023-10-06T01:40:10.547 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 119/122 2023-10-06T01:40:10.690 INFO:teuthology.orchestra.run.smithi027.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 120/122 2023-10-06T01:40:10.690 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-10-06T01:40:10.708 INFO:teuthology.orchestra.run.smithi124.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 122/122 2023-10-06T01:40:10.708 INFO:teuthology.orchestra.run.smithi124.stdout:Last metadata expiration check: 0:01:44 ago on Fri 06 Oct 2023 01:38:26 AM UTC. 2023-10-06T01:40:10.899 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-10-06T01:40:10.899 INFO:teuthology.orchestra.run.smithi027.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 122/122 2023-10-06T01:40:11.838 INFO:teuthology.orchestra.run.smithi027.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 122/122 2023-10-06T01:40:11.838 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-2:17.2.6-1296.g736ac5ba.el8.x86_64 1/122 2023-10-06T01:40:11.839 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-base-2:17.2.6-1296.g736ac5ba.el8.x86_64 2/122 2023-10-06T01:40:11.839 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-common-2:17.2.6-1296.g736ac5ba.el8.x86_64 3/122 2023-10-06T01:40:11.839 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-fuse-2:17.2.6-1296.g736ac5ba.el8.x86_64 4/122 2023-10-06T01:40:11.839 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-1296.g736ac 5/122 2023-10-06T01:40:11.839 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mds-2:17.2.6-1296.g736ac5ba.el8.x86_64 6/122 2023-10-06T01:40:11.839 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-2:17.2.6-1296.g736ac5ba.el8.x86_64 7/122 2023-10-06T01:40:11.839 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mon-2:17.2.6-1296.g736ac5ba.el8.x86_64 8/122 2023-10-06T01:40:11.840 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-osd-2:17.2.6-1296.g736ac5ba.el8.x86_64 9/122 2023-10-06T01:40:11.840 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-radosgw-2:17.2.6-1296.g736ac5ba.el8.x86_64 10/122 2023-10-06T01:40:11.840 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-selinux-2:17.2.6-1296.g736ac5ba.el8.x86_64 11/122 2023-10-06T01:40:11.840 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-test-2:17.2.6-1296.g736ac5ba.el8.x86_64 12/122 2023-10-06T01:40:11.840 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephfs-mirror-2:17.2.6-1296.g736ac5ba.el8.x86_64 13/122 2023-10-06T01:40:11.840 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs-devel-2:17.2.6-1296.g736ac5ba.el8.x86_ 14/122 2023-10-06T01:40:11.840 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephfs2-2:17.2.6-1296.g736ac5ba.el8.x86_64 15/122 2023-10-06T01:40:11.840 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcephsqlite-2:17.2.6-1296.g736ac5ba.el8.x86_64 16/122 2023-10-06T01:40:11.841 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados-devel-2:17.2.6-1296.g736ac5ba.el8.x86_6 17/122 2023-10-06T01:40:11.841 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libradosstriper1-2:17.2.6-1296.g736ac5ba.el8.x86 18/122 2023-10-06T01:40:11.841 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librgw2-2:17.2.6-1296.g736ac5ba.el8.x86_64 19/122 2023-10-06T01:40:11.841 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-argparse-2:17.2.6-1296.g736ac5ba.el 20/122 2023-10-06T01:40:11.841 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-ceph-common-2:17.2.6-1296.g736ac5ba.el8. 21/122 2023-10-06T01:40:11.841 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cephfs-2:17.2.6-1296.g736ac5ba.el8.x86_6 22/122 2023-10-06T01:40:11.842 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rados-2:17.2.6-1296.g736ac5ba.el8.x86_64 23/122 2023-10-06T01:40:11.842 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rbd-2:17.2.6-1296.g736ac5ba.el8.x86_64 24/122 2023-10-06T01:40:11.842 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rgw-2:17.2.6-1296.g736ac5ba.el8.x86_64 25/122 2023-10-06T01:40:11.842 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-fuse-2:17.2.6-1296.g736ac5ba.el8.x86_64 26/122 2023-10-06T01:40:11.842 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-mirror-2:17.2.6-1296.g736ac5ba.el8.x86_64 27/122 2023-10-06T01:40:11.842 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : rbd-nbd-2:17.2.6-1296.g736ac5ba.el8.x86_64 28/122 2023-10-06T01:40:11.842 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-1296.g736ac5ba. 29/122 2023-10-06T01:40:11.843 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-1296.g736ac5ba.el8.noa 30/122 2023-10-06T01:40:11.843 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-1296.g736ac5ba.el8.n 31/122 2023-10-06T01:40:11.843 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-1296.g736 32/122 2023-10-06T01:40:11.843 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-1296.g736ac5ba.el 33/122 2023-10-06T01:40:11.843 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-mgr-rook-2:17.2.6-1296.g736ac5ba.el8.noarch 34/122 2023-10-06T01:40:11.843 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-1296.g736ac5ba.e 35/122 2023-10-06T01:40:11.843 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : ceph-volume-2:17.2.6-1296.g736ac5ba.el8.noarch 36/122 2023-10-06T01:40:11.844 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephadm-2:17.2.6-1296.g736ac5ba.el8.noarch 37/122 2023-10-06T01:40:11.844 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : cephfs-top-2:17.2.6-1296.g736ac5ba.el8.noarch 38/122 2023-10-06T01:40:11.844 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/122 2023-10-06T01:40:11.844 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/122 2023-10-06T01:40:11.844 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/122 2023-10-06T01:40:11.844 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/122 2023-10-06T01:40:11.844 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/122 2023-10-06T01:40:11.844 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/122 2023-10-06T01:40:11.845 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/122 2023-10-06T01:40:11.845 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/122 2023-10-06T01:40:11.845 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/122 2023-10-06T01:40:11.845 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/122 2023-10-06T01:40:11.845 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/122 2023-10-06T01:40:11.845 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/122 2023-10-06T01:40:11.845 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/122 2023-10-06T01:40:11.845 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/122 2023-10-06T01:40:11.845 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/122 2023-10-06T01:40:11.845 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/122 2023-10-06T01:40:11.846 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/122 2023-10-06T01:40:11.846 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/122 2023-10-06T01:40:11.846 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/122 2023-10-06T01:40:11.846 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/122 2023-10-06T01:40:11.846 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/122 2023-10-06T01:40:11.846 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 60/122 2023-10-06T01:40:11.846 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/122 2023-10-06T01:40:11.846 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/122 2023-10-06T01:40:11.846 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/122 2023-10-06T01:40:11.846 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/122 2023-10-06T01:40:11.846 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/122 2023-10-06T01:40:11.846 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/122 2023-10-06T01:40:11.847 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/122 2023-10-06T01:40:11.847 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/122 2023-10-06T01:40:11.847 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/122 2023-10-06T01:40:11.847 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/122 2023-10-06T01:40:11.847 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/122 2023-10-06T01:40:11.847 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 72/122 2023-10-06T01:40:11.847 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 73/122 2023-10-06T01:40:11.847 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 74/122 2023-10-06T01:40:11.847 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/122 2023-10-06T01:40:11.847 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 76/122 2023-10-06T01:40:11.847 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 77/122 2023-10-06T01:40:11.847 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/122 2023-10-06T01:40:11.848 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openssl-devel-1:1.1.1g-15.el8_3.x86_64 79/122 2023-10-06T01:40:11.848 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 80/122 2023-10-06T01:40:11.848 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 81/122 2023-10-06T01:40:11.848 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 82/122 2023-10-06T01:40:11.848 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : elfutils-libelf-devel-0.182-3.el8.x86_64 83/122 2023-10-06T01:40:11.848 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 84/122 2023-10-06T01:40:11.848 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libcom_err-devel-1.45.6-1.el8.x86_64 85/122 2023-10-06T01:40:11.848 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libsepol-devel-2.9-2.el8.x86_64 86/122 2023-10-06T01:40:11.848 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 87/122 2023-10-06T01:40:11.848 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 88/122 2023-10-06T01:40:11.848 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : krb5-devel-1.18.2-8.3.el8_4.x86_64 89/122 2023-10-06T01:40:11.848 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 90/122 2023-10-06T01:40:11.848 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : xfsprogs-devel-5.0.0-8.el8.x86_64 91/122 2023-10-06T01:40:11.849 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 92/122 2023-10-06T01:40:11.849 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libuuid-devel-2.32.1-27.el8.x86_64 93/122 2023-10-06T01:40:11.849 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 94/122 2023-10-06T01:40:11.849 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 95/122 2023-10-06T01:40:11.849 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 96/122 2023-10-06T01:40:11.849 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : jq-1.5-12.el8.x86_64 97/122 2023-10-06T01:40:11.849 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : flex-2.6.1-9.el8.x86_64 98/122 2023-10-06T01:40:11.849 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 99/122 2023-10-06T01:40:11.849 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 100/122 2023-10-06T01:40:11.849 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 101/122 2023-10-06T01:40:11.849 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : bison-3.0.4-10.el8.x86_64 102/122 2023-10-06T01:40:11.849 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 103/122 2023-10-06T01:40:11.849 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 104/122 2023-10-06T01:40:11.850 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 105/122 2023-10-06T01:40:11.850 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 106/122 2023-10-06T01:40:11.850 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/122 2023-10-06T01:40:11.850 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 108/122 2023-10-06T01:40:11.850 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 109/122 2023-10-06T01:40:11.850 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 110/122 2023-10-06T01:40:11.850 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 111/122 2023-10-06T01:40:11.850 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 112/122 2023-10-06T01:40:11.850 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 113/122 2023-10-06T01:40:11.850 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 114/122 2023-10-06T01:40:11.850 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 115/122 2023-10-06T01:40:11.850 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 116/122 2023-10-06T01:40:11.851 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 117/122 2023-10-06T01:40:11.851 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : automake-1.16.1-7.el8.noarch 118/122 2023-10-06T01:40:11.851 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-2:17.2.6-1296.g736ac5ba.el8.x86_64 119/122 2023-10-06T01:40:11.851 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 120/122 2023-10-06T01:40:11.851 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-2:17.2.6-1296.g736ac5ba.el8.x86_64 121/122 2023-10-06T01:40:13.123 INFO:teuthology.orchestra.run.smithi124.stdout:Installed products updated. 2023-10-06T01:40:13.267 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-10-06T01:40:13.268 INFO:teuthology.orchestra.run.smithi124.stdout:Upgraded: 2023-10-06T01:40:13.268 INFO:teuthology.orchestra.run.smithi124.stdout: librados2-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:13.268 INFO:teuthology.orchestra.run.smithi124.stdout: librbd1-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:13.268 INFO:teuthology.orchestra.run.smithi124.stdout:Installed: 2023-10-06T01:40:13.268 INFO:teuthology.orchestra.run.smithi124.stdout: automake-1.16.1-7.el8.noarch 2023-10-06T01:40:13.268 INFO:teuthology.orchestra.run.smithi124.stdout: bison-3.0.4-10.el8.x86_64 2023-10-06T01:40:13.268 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:13.268 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-base-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:13.268 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-common-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:13.269 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-fuse-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:13.269 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-grafana-dashboards-2:17.2.6-1296.g736ac5ba.el8.noarch 2023-10-06T01:40:13.269 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-immutable-object-cache-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:13.269 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mds-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:13.269 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:13.269 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-cephadm-2:17.2.6-1296.g736ac5ba.el8.noarch 2023-10-06T01:40:13.269 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-dashboard-2:17.2.6-1296.g736ac5ba.el8.noarch 2023-10-06T01:40:13.269 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-diskprediction-local-2:17.2.6-1296.g736ac5ba.el8.noarch 2023-10-06T01:40:13.269 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-modules-core-2:17.2.6-1296.g736ac5ba.el8.noarch 2023-10-06T01:40:13.269 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-rook-2:17.2.6-1296.g736ac5ba.el8.noarch 2023-10-06T01:40:13.269 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mon-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:13.270 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-osd-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:13.270 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-prometheus-alerts-2:17.2.6-1296.g736ac5ba.el8.noarch 2023-10-06T01:40:13.270 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-radosgw-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:13.270 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-selinux-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:13.270 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-test-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:13.270 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-volume-2:17.2.6-1296.g736ac5ba.el8.noarch 2023-10-06T01:40:13.270 INFO:teuthology.orchestra.run.smithi124.stdout: cephadm-2:17.2.6-1296.g736ac5ba.el8.noarch 2023-10-06T01:40:13.270 INFO:teuthology.orchestra.run.smithi124.stdout: cephfs-mirror-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:13.270 INFO:teuthology.orchestra.run.smithi124.stdout: cephfs-top-2:17.2.6-1296.g736ac5ba.el8.noarch 2023-10-06T01:40:13.270 INFO:teuthology.orchestra.run.smithi124.stdout: elfutils-libelf-devel-0.182-3.el8.x86_64 2023-10-06T01:40:13.270 INFO:teuthology.orchestra.run.smithi124.stdout: flex-2.6.1-9.el8.x86_64 2023-10-06T01:40:13.271 INFO:teuthology.orchestra.run.smithi124.stdout: fuse-2.9.7-12.el8.x86_64 2023-10-06T01:40:13.271 INFO:teuthology.orchestra.run.smithi124.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-06T01:40:13.271 INFO:teuthology.orchestra.run.smithi124.stdout: jq-1.5-12.el8.x86_64 2023-10-06T01:40:13.271 INFO:teuthology.orchestra.run.smithi124.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2023-10-06T01:40:13.271 INFO:teuthology.orchestra.run.smithi124.stdout: krb5-devel-1.18.2-8.3.el8_4.x86_64 2023-10-06T01:40:13.271 INFO:teuthology.orchestra.run.smithi124.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-10-06T01:40:13.271 INFO:teuthology.orchestra.run.smithi124.stdout: libcephfs-devel-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:13.271 INFO:teuthology.orchestra.run.smithi124.stdout: libcephfs2-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:13.271 INFO:teuthology.orchestra.run.smithi124.stdout: libcephsqlite-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:13.271 INFO:teuthology.orchestra.run.smithi124.stdout: libcom_err-devel-1.45.6-1.el8.x86_64 2023-10-06T01:40:13.271 INFO:teuthology.orchestra.run.smithi124.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-10-06T01:40:13.271 INFO:teuthology.orchestra.run.smithi124.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-06T01:40:13.272 INFO:teuthology.orchestra.run.smithi124.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-10-06T01:40:13.272 INFO:teuthology.orchestra.run.smithi124.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-10-06T01:40:13.272 INFO:teuthology.orchestra.run.smithi124.stdout: librados-devel-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:13.272 INFO:teuthology.orchestra.run.smithi124.stdout: libradosstriper1-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:13.272 INFO:teuthology.orchestra.run.smithi124.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-10-06T01:40:13.272 INFO:teuthology.orchestra.run.smithi124.stdout: librgw2-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:13.272 INFO:teuthology.orchestra.run.smithi124.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-10-06T01:40:13.272 INFO:teuthology.orchestra.run.smithi124.stdout: libsepol-devel-2.9-2.el8.x86_64 2023-10-06T01:40:13.272 INFO:teuthology.orchestra.run.smithi124.stdout: libtool-2.4.6-25.el8.x86_64 2023-10-06T01:40:13.272 INFO:teuthology.orchestra.run.smithi124.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-06T01:40:13.272 INFO:teuthology.orchestra.run.smithi124.stdout: libuuid-devel-2.32.1-27.el8.x86_64 2023-10-06T01:40:13.272 INFO:teuthology.orchestra.run.smithi124.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-10-06T01:40:13.273 INFO:teuthology.orchestra.run.smithi124.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-06T01:40:13.273 INFO:teuthology.orchestra.run.smithi124.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-06T01:40:13.273 INFO:teuthology.orchestra.run.smithi124.stdout: openblas-0.3.12-1.el8.x86_64 2023-10-06T01:40:13.273 INFO:teuthology.orchestra.run.smithi124.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-10-06T01:40:13.273 INFO:teuthology.orchestra.run.smithi124.stdout: openssl-devel-1:1.1.1g-15.el8_3.x86_64 2023-10-06T01:40:13.273 INFO:teuthology.orchestra.run.smithi124.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-10-06T01:40:13.273 INFO:teuthology.orchestra.run.smithi124.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-10-06T01:40:13.273 INFO:teuthology.orchestra.run.smithi124.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-10-06T01:40:13.273 INFO:teuthology.orchestra.run.smithi124.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-10-06T01:40:13.273 INFO:teuthology.orchestra.run.smithi124.stdout: python3-babel-2.5.1-5.el8.noarch 2023-10-06T01:40:13.273 INFO:teuthology.orchestra.run.smithi124.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-06T01:40:13.273 INFO:teuthology.orchestra.run.smithi124.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-06T01:40:13.274 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-06T01:40:13.274 INFO:teuthology.orchestra.run.smithi124.stdout: python3-ceph-argparse-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:13.274 INFO:teuthology.orchestra.run.smithi124.stdout: python3-ceph-common-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:13.274 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cephfs-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:13.274 INFO:teuthology.orchestra.run.smithi124.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-06T01:40:13.274 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-06T01:40:13.274 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-06T01:40:13.274 INFO:teuthology.orchestra.run.smithi124.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-06T01:40:13.274 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-06T01:40:13.274 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-06T01:40:13.274 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-10-06T01:40:13.275 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-06T01:40:13.275 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-06T01:40:13.275 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-06T01:40:13.275 INFO:teuthology.orchestra.run.smithi124.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-06T01:40:13.275 INFO:teuthology.orchestra.run.smithi124.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-06T01:40:13.275 INFO:teuthology.orchestra.run.smithi124.stdout: python3-mako-1.0.6-13.el8.noarch 2023-10-06T01:40:13.275 INFO:teuthology.orchestra.run.smithi124.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-06T01:40:13.275 INFO:teuthology.orchestra.run.smithi124.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-06T01:40:13.275 INFO:teuthology.orchestra.run.smithi124.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-10-06T01:40:13.275 INFO:teuthology.orchestra.run.smithi124.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-10-06T01:40:13.275 INFO:teuthology.orchestra.run.smithi124.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-10-06T01:40:13.275 INFO:teuthology.orchestra.run.smithi124.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-06T01:40:13.276 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-06T01:40:13.276 INFO:teuthology.orchestra.run.smithi124.stdout: python3-portend-2.6-1.el8.noarch 2023-10-06T01:40:13.276 INFO:teuthology.orchestra.run.smithi124.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-06T01:40:13.276 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-06T01:40:13.276 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-06T01:40:13.276 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pytz-2017.2-9.el8.noarch 2023-10-06T01:40:13.276 INFO:teuthology.orchestra.run.smithi124.stdout: python3-rados-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:13.276 INFO:teuthology.orchestra.run.smithi124.stdout: python3-rbd-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:13.276 INFO:teuthology.orchestra.run.smithi124.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-06T01:40:13.276 INFO:teuthology.orchestra.run.smithi124.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-06T01:40:13.276 INFO:teuthology.orchestra.run.smithi124.stdout: python3-rgw-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:13.276 INFO:teuthology.orchestra.run.smithi124.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-06T01:40:13.277 INFO:teuthology.orchestra.run.smithi124.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-06T01:40:13.277 INFO:teuthology.orchestra.run.smithi124.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-10-06T01:40:13.277 INFO:teuthology.orchestra.run.smithi124.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-06T01:40:13.277 INFO:teuthology.orchestra.run.smithi124.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-06T01:40:13.277 INFO:teuthology.orchestra.run.smithi124.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-06T01:40:13.277 INFO:teuthology.orchestra.run.smithi124.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-06T01:40:13.277 INFO:teuthology.orchestra.run.smithi124.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-06T01:40:13.277 INFO:teuthology.orchestra.run.smithi124.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-06T01:40:13.277 INFO:teuthology.orchestra.run.smithi124.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-06T01:40:13.277 INFO:teuthology.orchestra.run.smithi124.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-06T01:40:13.277 INFO:teuthology.orchestra.run.smithi124.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-06T01:40:13.278 INFO:teuthology.orchestra.run.smithi124.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-06T01:40:13.278 INFO:teuthology.orchestra.run.smithi124.stdout: rbd-fuse-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:13.278 INFO:teuthology.orchestra.run.smithi124.stdout: rbd-mirror-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:13.278 INFO:teuthology.orchestra.run.smithi124.stdout: rbd-nbd-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:13.278 INFO:teuthology.orchestra.run.smithi124.stdout: socat-1.7.3.3-2.el8.x86_64 2023-10-06T01:40:13.278 INFO:teuthology.orchestra.run.smithi124.stdout: thrift-0.13.0-2.el8.x86_64 2023-10-06T01:40:13.278 INFO:teuthology.orchestra.run.smithi124.stdout: xfsprogs-devel-5.0.0-8.el8.x86_64 2023-10-06T01:40:13.278 INFO:teuthology.orchestra.run.smithi124.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-06T01:40:13.279 INFO:teuthology.orchestra.run.smithi124.stdout: zlib-devel-1.2.11-17.el8.x86_64 2023-10-06T01:40:13.279 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-10-06T01:40:13.279 INFO:teuthology.orchestra.run.smithi124.stdout:Complete! 2023-10-06T01:40:13.526 DEBUG:teuthology.parallel:result is None 2023-10-06T01:40:13.993 INFO:teuthology.orchestra.run.smithi027.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 122/122 2023-10-06T01:40:13.993 INFO:teuthology.orchestra.run.smithi027.stdout:Last metadata expiration check: 0:01:46 ago on Fri 06 Oct 2023 01:38:27 AM UTC. 2023-10-06T01:40:16.366 INFO:teuthology.orchestra.run.smithi027.stdout:Installed products updated. 2023-10-06T01:40:16.543 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:40:16.543 INFO:teuthology.orchestra.run.smithi027.stdout:Upgraded: 2023-10-06T01:40:16.543 INFO:teuthology.orchestra.run.smithi027.stdout: librados2-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:16.543 INFO:teuthology.orchestra.run.smithi027.stdout: librbd1-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:16.543 INFO:teuthology.orchestra.run.smithi027.stdout:Installed: 2023-10-06T01:40:16.543 INFO:teuthology.orchestra.run.smithi027.stdout: automake-1.16.1-7.el8.noarch 2023-10-06T01:40:16.543 INFO:teuthology.orchestra.run.smithi027.stdout: bison-3.0.4-10.el8.x86_64 2023-10-06T01:40:16.544 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:16.544 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-base-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:16.544 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-common-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:16.544 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-fuse-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:16.544 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-grafana-dashboards-2:17.2.6-1296.g736ac5ba.el8.noarch 2023-10-06T01:40:16.544 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-immutable-object-cache-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:16.544 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mds-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:16.544 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:16.544 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-cephadm-2:17.2.6-1296.g736ac5ba.el8.noarch 2023-10-06T01:40:16.544 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-dashboard-2:17.2.6-1296.g736ac5ba.el8.noarch 2023-10-06T01:40:16.544 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-diskprediction-local-2:17.2.6-1296.g736ac5ba.el8.noarch 2023-10-06T01:40:16.544 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-modules-core-2:17.2.6-1296.g736ac5ba.el8.noarch 2023-10-06T01:40:16.545 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mgr-rook-2:17.2.6-1296.g736ac5ba.el8.noarch 2023-10-06T01:40:16.545 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-mon-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:16.545 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-osd-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:16.545 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-prometheus-alerts-2:17.2.6-1296.g736ac5ba.el8.noarch 2023-10-06T01:40:16.545 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-radosgw-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:16.545 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-selinux-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:16.545 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-test-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:16.545 INFO:teuthology.orchestra.run.smithi027.stdout: ceph-volume-2:17.2.6-1296.g736ac5ba.el8.noarch 2023-10-06T01:40:16.545 INFO:teuthology.orchestra.run.smithi027.stdout: cephadm-2:17.2.6-1296.g736ac5ba.el8.noarch 2023-10-06T01:40:16.545 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-mirror-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:16.545 INFO:teuthology.orchestra.run.smithi027.stdout: cephfs-top-2:17.2.6-1296.g736ac5ba.el8.noarch 2023-10-06T01:40:16.545 INFO:teuthology.orchestra.run.smithi027.stdout: elfutils-libelf-devel-0.182-3.el8.x86_64 2023-10-06T01:40:16.546 INFO:teuthology.orchestra.run.smithi027.stdout: flex-2.6.1-9.el8.x86_64 2023-10-06T01:40:16.546 INFO:teuthology.orchestra.run.smithi027.stdout: fuse-2.9.7-12.el8.x86_64 2023-10-06T01:40:16.546 INFO:teuthology.orchestra.run.smithi027.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-10-06T01:40:16.546 INFO:teuthology.orchestra.run.smithi027.stdout: jq-1.5-12.el8.x86_64 2023-10-06T01:40:16.546 INFO:teuthology.orchestra.run.smithi027.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2023-10-06T01:40:16.546 INFO:teuthology.orchestra.run.smithi027.stdout: krb5-devel-1.18.2-8.3.el8_4.x86_64 2023-10-06T01:40:16.546 INFO:teuthology.orchestra.run.smithi027.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-10-06T01:40:16.546 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs-devel-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:16.546 INFO:teuthology.orchestra.run.smithi027.stdout: libcephfs2-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:16.546 INFO:teuthology.orchestra.run.smithi027.stdout: libcephsqlite-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:16.546 INFO:teuthology.orchestra.run.smithi027.stdout: libcom_err-devel-1.45.6-1.el8.x86_64 2023-10-06T01:40:16.546 INFO:teuthology.orchestra.run.smithi027.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-10-06T01:40:16.547 INFO:teuthology.orchestra.run.smithi027.stdout: liboath-2.6.2-3.el8.x86_64 2023-10-06T01:40:16.547 INFO:teuthology.orchestra.run.smithi027.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-10-06T01:40:16.547 INFO:teuthology.orchestra.run.smithi027.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-10-06T01:40:16.547 INFO:teuthology.orchestra.run.smithi027.stdout: librados-devel-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:16.547 INFO:teuthology.orchestra.run.smithi027.stdout: libradosstriper1-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:16.547 INFO:teuthology.orchestra.run.smithi027.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-10-06T01:40:16.547 INFO:teuthology.orchestra.run.smithi027.stdout: librgw2-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:16.547 INFO:teuthology.orchestra.run.smithi027.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-10-06T01:40:16.547 INFO:teuthology.orchestra.run.smithi027.stdout: libsepol-devel-2.9-2.el8.x86_64 2023-10-06T01:40:16.547 INFO:teuthology.orchestra.run.smithi027.stdout: libtool-2.4.6-25.el8.x86_64 2023-10-06T01:40:16.547 INFO:teuthology.orchestra.run.smithi027.stdout: libunwind-1.3.1-3.el8.x86_64 2023-10-06T01:40:16.547 INFO:teuthology.orchestra.run.smithi027.stdout: libuuid-devel-2.32.1-27.el8.x86_64 2023-10-06T01:40:16.548 INFO:teuthology.orchestra.run.smithi027.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-10-06T01:40:16.548 INFO:teuthology.orchestra.run.smithi027.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-10-06T01:40:16.548 INFO:teuthology.orchestra.run.smithi027.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-10-06T01:40:16.548 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-0.3.12-1.el8.x86_64 2023-10-06T01:40:16.548 INFO:teuthology.orchestra.run.smithi027.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-10-06T01:40:16.548 INFO:teuthology.orchestra.run.smithi027.stdout: openssl-devel-1:1.1.1g-15.el8_3.x86_64 2023-10-06T01:40:16.548 INFO:teuthology.orchestra.run.smithi027.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-10-06T01:40:16.548 INFO:teuthology.orchestra.run.smithi027.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-10-06T01:40:16.548 INFO:teuthology.orchestra.run.smithi027.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-10-06T01:40:16.548 INFO:teuthology.orchestra.run.smithi027.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-10-06T01:40:16.548 INFO:teuthology.orchestra.run.smithi027.stdout: python3-babel-2.5.1-5.el8.noarch 2023-10-06T01:40:16.549 INFO:teuthology.orchestra.run.smithi027.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-10-06T01:40:16.549 INFO:teuthology.orchestra.run.smithi027.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-10-06T01:40:16.549 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-10-06T01:40:16.549 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-argparse-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:16.549 INFO:teuthology.orchestra.run.smithi027.stdout: python3-ceph-common-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:16.549 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cephfs-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:16.549 INFO:teuthology.orchestra.run.smithi027.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-10-06T01:40:16.549 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-10-06T01:40:16.549 INFO:teuthology.orchestra.run.smithi027.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-10-06T01:40:16.549 INFO:teuthology.orchestra.run.smithi027.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-10-06T01:40:16.549 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-6.2-6.el8.noarch 2023-10-06T01:40:16.549 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-10-06T01:40:16.550 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-10-06T01:40:16.550 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-10-06T01:40:16.550 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-10-06T01:40:16.550 INFO:teuthology.orchestra.run.smithi027.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-10-06T01:40:16.550 INFO:teuthology.orchestra.run.smithi027.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-10-06T01:40:16.550 INFO:teuthology.orchestra.run.smithi027.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-10-06T01:40:16.550 INFO:teuthology.orchestra.run.smithi027.stdout: python3-mako-1.0.6-13.el8.noarch 2023-10-06T01:40:16.550 INFO:teuthology.orchestra.run.smithi027.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-10-06T01:40:16.550 INFO:teuthology.orchestra.run.smithi027.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-10-06T01:40:16.550 INFO:teuthology.orchestra.run.smithi027.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-10-06T01:40:16.550 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-10-06T01:40:16.550 INFO:teuthology.orchestra.run.smithi027.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-10-06T01:40:16.551 INFO:teuthology.orchestra.run.smithi027.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-10-06T01:40:16.551 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-10-06T01:40:16.551 INFO:teuthology.orchestra.run.smithi027.stdout: python3-portend-2.6-1.el8.noarch 2023-10-06T01:40:16.551 INFO:teuthology.orchestra.run.smithi027.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-10-06T01:40:16.551 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-10-06T01:40:16.551 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-10-06T01:40:16.551 INFO:teuthology.orchestra.run.smithi027.stdout: python3-pytz-2017.2-9.el8.noarch 2023-10-06T01:40:16.551 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rados-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:16.551 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rbd-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:16.552 INFO:teuthology.orchestra.run.smithi027.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-10-06T01:40:16.552 INFO:teuthology.orchestra.run.smithi027.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-10-06T01:40:16.552 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rgw-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:16.552 INFO:teuthology.orchestra.run.smithi027.stdout: python3-routes-2.4.1-12.el8.noarch 2023-10-06T01:40:16.552 INFO:teuthology.orchestra.run.smithi027.stdout: python3-rsa-4.9-2.el8.noarch 2023-10-06T01:40:16.552 INFO:teuthology.orchestra.run.smithi027.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-10-06T01:40:16.552 INFO:teuthology.orchestra.run.smithi027.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-10-06T01:40:16.552 INFO:teuthology.orchestra.run.smithi027.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-10-06T01:40:16.552 INFO:teuthology.orchestra.run.smithi027.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-10-06T01:40:16.552 INFO:teuthology.orchestra.run.smithi027.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-10-06T01:40:16.552 INFO:teuthology.orchestra.run.smithi027.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-10-06T01:40:16.553 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-10-06T01:40:16.553 INFO:teuthology.orchestra.run.smithi027.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-10-06T01:40:16.553 INFO:teuthology.orchestra.run.smithi027.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-10-06T01:40:16.553 INFO:teuthology.orchestra.run.smithi027.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-10-06T01:40:16.553 INFO:teuthology.orchestra.run.smithi027.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-10-06T01:40:16.553 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-fuse-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:16.553 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-mirror-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:16.553 INFO:teuthology.orchestra.run.smithi027.stdout: rbd-nbd-2:17.2.6-1296.g736ac5ba.el8.x86_64 2023-10-06T01:40:16.553 INFO:teuthology.orchestra.run.smithi027.stdout: socat-1.7.3.3-2.el8.x86_64 2023-10-06T01:40:16.553 INFO:teuthology.orchestra.run.smithi027.stdout: thrift-0.13.0-2.el8.x86_64 2023-10-06T01:40:16.553 INFO:teuthology.orchestra.run.smithi027.stdout: xfsprogs-devel-5.0.0-8.el8.x86_64 2023-10-06T01:40:16.553 INFO:teuthology.orchestra.run.smithi027.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-10-06T01:40:16.554 INFO:teuthology.orchestra.run.smithi027.stdout: zlib-devel-1.2.11-17.el8.x86_64 2023-10-06T01:40:16.554 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:40:16.554 INFO:teuthology.orchestra.run.smithi027.stdout:Complete! 2023-10-06T01:40:16.813 DEBUG:teuthology.parallel:result is None 2023-10-06T01:40:16.813 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=736ac5ba72cb47af7d6f6666b1e100f7e20a20e2 2023-10-06T01:40:17.035 DEBUG:teuthology.orchestra.run.smithi027:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-06T01:40:17.057 INFO:teuthology.orchestra.run.smithi027.stdout:17.2.6-1296.g736ac5ba.el8 2023-10-06T01:40:17.057 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1296.g736ac5ba.el8 2023-10-06T01:40:17.057 INFO:teuthology.task.install:The correct ceph version 17.2.6-1296.g736ac5ba is installed. 2023-10-06T01:40:17.058 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=736ac5ba72cb47af7d6f6666b1e100f7e20a20e2 2023-10-06T01:40:17.198 DEBUG:teuthology.orchestra.run.smithi124:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-06T01:40:17.222 INFO:teuthology.orchestra.run.smithi124.stdout:17.2.6-1296.g736ac5ba.el8 2023-10-06T01:40:17.222 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1296.g736ac5ba.el8 2023-10-06T01:40:17.222 INFO:teuthology.task.install:The correct ceph version 17.2.6-1296.g736ac5ba is installed. 2023-10-06T01:40:17.223 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-06T01:40:17.223 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-06T01:40:17.223 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-06T01:40:17.263 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-10-06T01:40:17.263 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-06T01:40:17.304 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-06T01:40:17.305 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-06T01:40:17.305 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/daemon-helper 2023-10-06T01:40:17.331 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-06T01:40:17.396 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-10-06T01:40:17.396 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd of=/usr/bin/daemon-helper 2023-10-06T01:40:17.422 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-06T01:40:17.489 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-06T01:40:17.489 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-06T01:40:17.489 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-06T01:40:17.513 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-06T01:40:17.577 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-10-06T01:40:17.577 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-06T01:40:17.603 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-06T01:40:17.667 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-06T01:40:17.667 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-06T01:40:17.667 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/usr/bin/stdin-killer 2023-10-06T01:40:17.691 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-06T01:40:17.758 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-10-06T01:40:17.758 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd of=/usr/bin/stdin-killer 2023-10-06T01:40:17.783 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-06T01:40:17.847 INFO:teuthology.run_tasks:Running task ceph... 2023-10-06T01:40:17.908 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-06T01:40:17.908 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 777 /var/log/ceph 2023-10-06T01:40:17.910 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod 777 /var/log/ceph 2023-10-06T01:40:17.934 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-06T01:40:17.934 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-06T01:40:17.976 DEBUG:teuthology.orchestra.run.smithi124:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-06T01:40:18.000 INFO:tasks.ceph:Creating extra log directories... 2023-10-06T01:40:18.000 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-06T01:40:18.041 DEBUG:teuthology.orchestra.run.smithi124:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-06T01:40:18.071 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-10-06T01:40:18.072 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-10-06T01:40:18.072 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-10-06T01:40:18.072 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-06T01:40:18.072 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-06T01:40:18.119 DEBUG:teuthology.orchestra.run.smithi027:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-06T01:40:18.230 DEBUG:teuthology.orchestra.run.smithi027:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-10-06T01:40:18.255 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-10-06T01:40:18.255 DEBUG:teuthology.orchestra.run.smithi124:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-06T01:40:18.269 DEBUG:teuthology.orchestra.run.smithi124:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-06T01:40:18.379 DEBUG:teuthology.orchestra.run.smithi124:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-10-06T01:40:18.409 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-06T01:40:18.409 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-06T01:40:18.409 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi/7412096', 'branch': 'wip-yuri-testing-2023-10-05-0730-quincy', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7412096', 'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'rhel', 'os_version': '8.4', 'overrides': {'admin_socket': {'branch': 'wip-yuri-testing-2023-10-05-0730-quincy'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '736ac5ba72cb47af7d6f6666b1e100f7e20a20e2'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '736ac5ba72cb47af7d6f6666b1e100f7e20a20e2'}}, 'kclient': {'syntax': 'v1'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri-testing-2023-10-05-0730-quincy', 'sha1': 'f1e315a9732d36c05ec316678e2397dc58561c7f'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 7519, 'sha1': '736ac5ba72cb47af7d6f6666b1e100f7e20a20e2', 'sleep_before_teardown': 0, 'subset': '111/12000', 'suite': 'fs', 'suite_branch': 'wip-yuri-testing-2023-10-05-0730-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_f1e315a9732d36c05ec316678e2397dc58561c7f/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'f1e315a9732d36c05ec316678e2397dc58561c7f', 'targets': {'smithi027.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKMVItaKZieNJGHWDSdhW5L1jqh1+xgxATt847HawcIgu7996RoKJqcfPy6oEBctp/7TfmFHn08ST1pb4RT6P/M=', 'smithi124.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPlciKcQZmKpVk5mhkr/65Bra2WtrkfGj0rG/nL0nqbOlucdiZdD+gY5t7TaSup1xruJNSuNkeYmPg9GEdBjrmo='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '736ac5ba72cb47af7d6f6666b1e100f7e20a20e2'}}, {'ceph': None}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_backtrace']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-10-05_23:52:50', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-06T01:40:18.410 DEBUG:teuthology.orchestra.run.smithi027:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-06T01:40:18.412 DEBUG:teuthology.orchestra.run.smithi124:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-06T01:40:18.458 DEBUG:teuthology.orchestra.run.smithi027:> sudo install -d -m0777 -- /var/run/ceph 2023-10-06T01:40:18.470 DEBUG:teuthology.orchestra.run.smithi124:> sudo install -d -m0777 -- /var/run/ceph 2023-10-06T01:40:18.523 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-06T01:40:18.523 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/scratch_devs of=/dev/stdout 2023-10-06T01:40:18.545 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-06T01:40:18.545 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_1 2023-10-06T01:40:18.600 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-06T01:40:18.600 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-06T01:40:18.600 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 6h/6d Inode: 214684 Links: 1 2023-10-06T01:40:18.600 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-06T01:40:18.601 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2023-10-06T01:40:18.601 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2023-10-06 01:40:11.364826091 +0000 2023-10-06T01:40:18.601 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2023-10-06 01:37:41.922682712 +0000 2023-10-06T01:40:18.601 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2023-10-06 01:37:41.922682712 +0000 2023-10-06T01:40:18.601 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: - 2023-10-06T01:40:18.601 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-06T01:40:18.663 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2023-10-06T01:40:18.663 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2023-10-06T01:40:18.663 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000247994 s, 2.1 MB/s 2023-10-06T01:40:18.664 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-06T01:40:18.721 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_2 2023-10-06T01:40:18.776 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-06T01:40:18.776 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-06T01:40:18.776 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 6h/6d Inode: 215825 Links: 1 2023-10-06T01:40:18.777 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-06T01:40:18.777 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2023-10-06T01:40:18.777 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2023-10-06 01:40:11.364826091 +0000 2023-10-06T01:40:18.777 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2023-10-06 01:37:42.423670224 +0000 2023-10-06T01:40:18.777 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2023-10-06 01:37:42.423670224 +0000 2023-10-06T01:40:18.777 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: - 2023-10-06T01:40:18.777 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-06T01:40:18.839 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2023-10-06T01:40:18.840 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2023-10-06T01:40:18.840 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000220941 s, 2.3 MB/s 2023-10-06T01:40:18.841 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-06T01:40:18.897 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_3 2023-10-06T01:40:18.952 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-06T01:40:18.952 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-06T01:40:18.952 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 6h/6d Inode: 217952 Links: 1 2023-10-06T01:40:18.953 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-06T01:40:18.953 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2023-10-06T01:40:18.953 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2023-10-06 01:40:11.364826091 +0000 2023-10-06T01:40:18.953 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2023-10-06 01:37:42.915657962 +0000 2023-10-06T01:40:18.953 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2023-10-06 01:37:42.915657962 +0000 2023-10-06T01:40:18.953 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: - 2023-10-06T01:40:18.953 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-06T01:40:19.016 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2023-10-06T01:40:19.016 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2023-10-06T01:40:19.016 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000243595 s, 2.1 MB/s 2023-10-06T01:40:19.017 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-06T01:40:19.073 DEBUG:teuthology.orchestra.run.smithi027:> stat /dev/vg_nvme/lv_4 2023-10-06T01:40:19.129 INFO:teuthology.orchestra.run.smithi027.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-06T01:40:19.130 INFO:teuthology.orchestra.run.smithi027.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-06T01:40:19.130 INFO:teuthology.orchestra.run.smithi027.stdout:Device: 6h/6d Inode: 212869 Links: 1 2023-10-06T01:40:19.130 INFO:teuthology.orchestra.run.smithi027.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-06T01:40:19.130 INFO:teuthology.orchestra.run.smithi027.stdout:Context: system_u:object_r:device_t:s0 2023-10-06T01:40:19.130 INFO:teuthology.orchestra.run.smithi027.stdout:Access: 2023-10-06 01:40:11.364826091 +0000 2023-10-06T01:40:19.130 INFO:teuthology.orchestra.run.smithi027.stdout:Modify: 2023-10-06 01:37:43.407645699 +0000 2023-10-06T01:40:19.130 INFO:teuthology.orchestra.run.smithi027.stdout:Change: 2023-10-06 01:37:43.407645699 +0000 2023-10-06T01:40:19.130 INFO:teuthology.orchestra.run.smithi027.stdout: Birth: - 2023-10-06T01:40:19.130 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-06T01:40:19.194 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records in 2023-10-06T01:40:19.194 INFO:teuthology.orchestra.run.smithi027.stderr:1+0 records out 2023-10-06T01:40:19.195 INFO:teuthology.orchestra.run.smithi027.stderr:512 bytes copied, 0.000192677 s, 2.7 MB/s 2023-10-06T01:40:19.196 DEBUG:teuthology.orchestra.run.smithi027:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-06T01:40:19.252 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-06T01:40:19.252 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-10-06T01:40:19.252 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/scratch_devs of=/dev/stdout 2023-10-06T01:40:19.268 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-06T01:40:19.268 DEBUG:teuthology.orchestra.run.smithi124:> stat /dev/vg_nvme/lv_1 2023-10-06T01:40:19.323 INFO:teuthology.orchestra.run.smithi124.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-06T01:40:19.324 INFO:teuthology.orchestra.run.smithi124.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-06T01:40:19.324 INFO:teuthology.orchestra.run.smithi124.stdout:Device: 6h/6d Inode: 217836 Links: 1 2023-10-06T01:40:19.324 INFO:teuthology.orchestra.run.smithi124.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-06T01:40:19.324 INFO:teuthology.orchestra.run.smithi124.stdout:Context: system_u:object_r:device_t:s0 2023-10-06T01:40:19.324 INFO:teuthology.orchestra.run.smithi124.stdout:Access: 2023-10-06 01:40:08.202299851 +0000 2023-10-06T01:40:19.324 INFO:teuthology.orchestra.run.smithi124.stdout:Modify: 2023-10-06 01:37:39.597578730 +0000 2023-10-06T01:40:19.324 INFO:teuthology.orchestra.run.smithi124.stdout:Change: 2023-10-06 01:37:39.597578730 +0000 2023-10-06T01:40:19.324 INFO:teuthology.orchestra.run.smithi124.stdout: Birth: - 2023-10-06T01:40:19.324 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-06T01:40:19.387 INFO:teuthology.orchestra.run.smithi124.stderr:1+0 records in 2023-10-06T01:40:19.388 INFO:teuthology.orchestra.run.smithi124.stderr:1+0 records out 2023-10-06T01:40:19.388 INFO:teuthology.orchestra.run.smithi124.stderr:512 bytes copied, 0.00025506 s, 2.0 MB/s 2023-10-06T01:40:19.389 DEBUG:teuthology.orchestra.run.smithi124:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-06T01:40:19.445 DEBUG:teuthology.orchestra.run.smithi124:> stat /dev/vg_nvme/lv_2 2023-10-06T01:40:19.501 INFO:teuthology.orchestra.run.smithi124.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-06T01:40:19.501 INFO:teuthology.orchestra.run.smithi124.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-06T01:40:19.501 INFO:teuthology.orchestra.run.smithi124.stdout:Device: 6h/6d Inode: 220460 Links: 1 2023-10-06T01:40:19.501 INFO:teuthology.orchestra.run.smithi124.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-06T01:40:19.501 INFO:teuthology.orchestra.run.smithi124.stdout:Context: system_u:object_r:device_t:s0 2023-10-06T01:40:19.501 INFO:teuthology.orchestra.run.smithi124.stdout:Access: 2023-10-06 01:40:08.203299834 +0000 2023-10-06T01:40:19.502 INFO:teuthology.orchestra.run.smithi124.stdout:Modify: 2023-10-06 01:37:40.088570392 +0000 2023-10-06T01:40:19.502 INFO:teuthology.orchestra.run.smithi124.stdout:Change: 2023-10-06 01:37:40.088570392 +0000 2023-10-06T01:40:19.502 INFO:teuthology.orchestra.run.smithi124.stdout: Birth: - 2023-10-06T01:40:19.502 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-06T01:40:19.564 INFO:teuthology.orchestra.run.smithi124.stderr:1+0 records in 2023-10-06T01:40:19.564 INFO:teuthology.orchestra.run.smithi124.stderr:1+0 records out 2023-10-06T01:40:19.564 INFO:teuthology.orchestra.run.smithi124.stderr:512 bytes copied, 0.000316479 s, 1.6 MB/s 2023-10-06T01:40:19.565 DEBUG:teuthology.orchestra.run.smithi124:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-06T01:40:19.620 DEBUG:teuthology.orchestra.run.smithi124:> stat /dev/vg_nvme/lv_3 2023-10-06T01:40:19.675 INFO:teuthology.orchestra.run.smithi124.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-06T01:40:19.675 INFO:teuthology.orchestra.run.smithi124.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-06T01:40:19.675 INFO:teuthology.orchestra.run.smithi124.stdout:Device: 6h/6d Inode: 218759 Links: 1 2023-10-06T01:40:19.675 INFO:teuthology.orchestra.run.smithi124.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-06T01:40:19.675 INFO:teuthology.orchestra.run.smithi124.stdout:Context: system_u:object_r:device_t:s0 2023-10-06T01:40:19.675 INFO:teuthology.orchestra.run.smithi124.stdout:Access: 2023-10-06 01:40:08.203299834 +0000 2023-10-06T01:40:19.675 INFO:teuthology.orchestra.run.smithi124.stdout:Modify: 2023-10-06 01:37:40.564562308 +0000 2023-10-06T01:40:19.675 INFO:teuthology.orchestra.run.smithi124.stdout:Change: 2023-10-06 01:37:40.564562308 +0000 2023-10-06T01:40:19.676 INFO:teuthology.orchestra.run.smithi124.stdout: Birth: - 2023-10-06T01:40:19.676 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-06T01:40:19.738 INFO:teuthology.orchestra.run.smithi124.stderr:1+0 records in 2023-10-06T01:40:19.738 INFO:teuthology.orchestra.run.smithi124.stderr:1+0 records out 2023-10-06T01:40:19.738 INFO:teuthology.orchestra.run.smithi124.stderr:512 bytes copied, 0.000226514 s, 2.3 MB/s 2023-10-06T01:40:19.739 DEBUG:teuthology.orchestra.run.smithi124:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-06T01:40:19.794 DEBUG:teuthology.orchestra.run.smithi124:> stat /dev/vg_nvme/lv_4 2023-10-06T01:40:19.850 INFO:teuthology.orchestra.run.smithi124.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-06T01:40:19.850 INFO:teuthology.orchestra.run.smithi124.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-06T01:40:19.850 INFO:teuthology.orchestra.run.smithi124.stdout:Device: 6h/6d Inode: 219844 Links: 1 2023-10-06T01:40:19.850 INFO:teuthology.orchestra.run.smithi124.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-06T01:40:19.850 INFO:teuthology.orchestra.run.smithi124.stdout:Context: system_u:object_r:device_t:s0 2023-10-06T01:40:19.850 INFO:teuthology.orchestra.run.smithi124.stdout:Access: 2023-10-06 01:40:08.203299834 +0000 2023-10-06T01:40:19.850 INFO:teuthology.orchestra.run.smithi124.stdout:Modify: 2023-10-06 01:37:41.056553953 +0000 2023-10-06T01:40:19.851 INFO:teuthology.orchestra.run.smithi124.stdout:Change: 2023-10-06 01:37:41.056553953 +0000 2023-10-06T01:40:19.851 INFO:teuthology.orchestra.run.smithi124.stdout: Birth: - 2023-10-06T01:40:19.851 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-06T01:40:19.913 INFO:teuthology.orchestra.run.smithi124.stderr:1+0 records in 2023-10-06T01:40:19.913 INFO:teuthology.orchestra.run.smithi124.stderr:1+0 records out 2023-10-06T01:40:19.913 INFO:teuthology.orchestra.run.smithi124.stderr:512 bytes copied, 0.000202655 s, 2.5 MB/s 2023-10-06T01:40:19.915 DEBUG:teuthology.orchestra.run.smithi124:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-06T01:40:19.970 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-06T01:40:19.971 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi027.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi124.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-06T01:40:19.971 INFO:tasks.ceph:Generating config... 2023-10-06T01:40:19.972 INFO:tasks.ceph:[client] client mount timeout = 600 2023-10-06T01:40:19.972 INFO:tasks.ceph:[client] debug client = 20 2023-10-06T01:40:19.972 INFO:tasks.ceph:[client] debug ms = 1 2023-10-06T01:40:19.973 INFO:tasks.ceph:[client] pid file = 2023-10-06T01:40:19.973 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-10-06T01:40:19.973 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-10-06T01:40:19.973 INFO:tasks.ceph:[global] ms die on skipped message = False 2023-10-06T01:40:19.973 INFO:tasks.ceph:[mds] debug mds = 20 2023-10-06T01:40:19.973 INFO:tasks.ceph:[mds] debug ms = 1 2023-10-06T01:40:19.973 INFO:tasks.ceph:[mds] mds debug frag = True 2023-10-06T01:40:19.973 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-10-06T01:40:19.974 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-10-06T01:40:19.974 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-10-06T01:40:19.974 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-10-06T01:40:19.974 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-10-06T01:40:19.974 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-10-06T01:40:19.974 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-06T01:40:19.974 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-06T01:40:19.974 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-06T01:40:19.975 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-06T01:40:19.975 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-06T01:40:19.975 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-10-06T01:40:19.975 INFO:tasks.ceph:[osd] bdev async discard = True 2023-10-06T01:40:19.975 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-10-06T01:40:19.975 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-10-06T01:40:19.975 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-06T01:40:19.975 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-06T01:40:19.975 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-10-06T01:40:19.975 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-10-06T01:40:19.975 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-06T01:40:19.975 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-06T01:40:19.976 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-10-06T01:40:19.976 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-06T01:40:19.976 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-06T01:40:19.976 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-06T01:40:19.976 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-06T01:40:19.976 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-06T01:40:19.976 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-10-06T01:40:19.976 INFO:tasks.ceph:Setting up mon.a... 2023-10-06T01:40:19.976 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-06T01:40:20.026 INFO:teuthology.orchestra.run.smithi027.stdout:creating /etc/ceph/ceph.keyring 2023-10-06T01:40:20.029 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-06T01:40:20.071 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-06T01:40:20.138 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.27'), ('mon.b', '172.21.15.124'), ('mon.c', '[v2:172.21.15.124:3301,v1:172.21.15.124:6790]')] 2023-10-06T01:40:20.139 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.27,172.21.15.124,[v2:172.21.15.124:3301,v1:172.21.15.124:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-10-06T01:40:20.140 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-06T01:40:20.140 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-06T01:40:20.193 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.27 --add b 172.21.15.124 --addv c '[v2:172.21.15.124:3301,v1:172.21.15.124:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-10-06T01:40:20.276 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-06T01:40:20.276 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool: generated fsid 20a7bff0-74f8-4f2f-b6c6-899e804bc965 2023-10-06T01:40:20.276 INFO:teuthology.orchestra.run.smithi027.stdout:setting min_mon_release = octopus 2023-10-06T01:40:20.276 INFO:teuthology.orchestra.run.smithi027.stdout:epoch 0 2023-10-06T01:40:20.276 INFO:teuthology.orchestra.run.smithi027.stdout:fsid 20a7bff0-74f8-4f2f-b6c6-899e804bc965 2023-10-06T01:40:20.277 INFO:teuthology.orchestra.run.smithi027.stdout:last_changed 2023-10-06T01:40:20.276614+0000 2023-10-06T01:40:20.277 INFO:teuthology.orchestra.run.smithi027.stdout:created 2023-10-06T01:40:20.276614+0000 2023-10-06T01:40:20.277 INFO:teuthology.orchestra.run.smithi027.stdout:min_mon_release 15 (octopus) 2023-10-06T01:40:20.277 INFO:teuthology.orchestra.run.smithi027.stdout:election_strategy: 1 2023-10-06T01:40:20.277 INFO:teuthology.orchestra.run.smithi027.stdout:0: [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] mon.a 2023-10-06T01:40:20.277 INFO:teuthology.orchestra.run.smithi027.stdout:1: [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] mon.b 2023-10-06T01:40:20.277 INFO:teuthology.orchestra.run.smithi027.stdout:2: [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] mon.c 2023-10-06T01:40:20.277 INFO:teuthology.orchestra.run.smithi027.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-10-06T01:40:20.278 DEBUG:teuthology.orchestra.run.smithi027:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-06T01:40:20.333 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 20a7bff0-74f8-4f2f-b6c6-899e804bc965... 2023-10-06T01:40:20.334 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-06T01:40:20.375 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-06T01:40:20.417 INFO:teuthology.orchestra.run.smithi027.stdout:[global] 2023-10-06T01:40:20.417 INFO:teuthology.orchestra.run.smithi027.stdout: chdir = "" 2023-10-06T01:40:20.418 INFO:teuthology.orchestra.run.smithi027.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-06T01:40:20.418 INFO:teuthology.orchestra.run.smithi027.stdout: auth supported = cephx 2023-10-06T01:40:20.418 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:40:20.418 INFO:teuthology.orchestra.run.smithi027.stdout: filestore xattr use omap = true 2023-10-06T01:40:20.418 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:40:20.418 INFO:teuthology.orchestra.run.smithi027.stdout: mon clock drift allowed = 1.000 2023-10-06T01:40:20.418 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:40:20.418 INFO:teuthology.orchestra.run.smithi027.stdout: osd crush chooseleaf type = 0 2023-10-06T01:40:20.418 INFO:teuthology.orchestra.run.smithi027.stdout: auth debug = true 2023-10-06T01:40:20.418 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:40:20.418 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on old message = true 2023-10-06T01:40:20.419 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on bug = true 2023-10-06T01:40:20.419 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:40:20.419 INFO:teuthology.orchestra.run.smithi027.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-06T01:40:20.419 INFO:teuthology.orchestra.run.smithi027.stdout: mon pg warn max object skew = 0 2023-10-06T01:40:20.419 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:40:20.419 INFO:teuthology.orchestra.run.smithi027.stdout: # disable pg_autoscaler by default for new pools 2023-10-06T01:40:20.419 INFO:teuthology.orchestra.run.smithi027.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-06T01:40:20.419 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:40:20.419 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default size = 2 2023-10-06T01:40:20.419 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:40:20.419 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow primary affinity = true 2023-10-06T01:40:20.419 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd allow pg remap = true 2023-10-06T01:40:20.419 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on legacy crush tunables = false 2023-10-06T01:40:20.420 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on crush straw calc version zero = false 2023-10-06T01:40:20.420 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on no sortbitwise = false 2023-10-06T01:40:20.420 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on osd down out interval zero = false 2023-10-06T01:40:20.420 INFO:teuthology.orchestra.run.smithi027.stdout: mon warn on too few osds = false 2023-10-06T01:40:20.420 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-06T01:40:20.420 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-06T01:40:20.420 INFO:teuthology.orchestra.run.smithi027.stdout: mon_allow_pool_size_one = true 2023-10-06T01:40:20.420 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:40:20.420 INFO:teuthology.orchestra.run.smithi027.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-06T01:40:20.420 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:40:20.420 INFO:teuthology.orchestra.run.smithi027.stdout: osd default data pool replay window = 5 2023-10-06T01:40:20.420 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:40:20.421 INFO:teuthology.orchestra.run.smithi027.stdout: mon allow pool delete = true 2023-10-06T01:40:20.421 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:40:20.421 INFO:teuthology.orchestra.run.smithi027.stdout: mon cluster log file level = debug 2023-10-06T01:40:20.421 INFO:teuthology.orchestra.run.smithi027.stdout: debug asserts on shutdown = true 2023-10-06T01:40:20.421 INFO:teuthology.orchestra.run.smithi027.stdout: mon health detail to clog = false 2023-10-06T01:40:20.421 INFO:teuthology.orchestra.run.smithi027.stdout: mon host = "172.21.15.27,172.21.15.124,[v2:172.21.15.124:3301,v1:172.21.15.124:6790]" 2023-10-06T01:40:20.421 INFO:teuthology.orchestra.run.smithi027.stdout: ms die on skipped message = False 2023-10-06T01:40:20.421 INFO:teuthology.orchestra.run.smithi027.stdout: fsid = 20a7bff0-74f8-4f2f-b6c6-899e804bc965 2023-10-06T01:40:20.421 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:40:20.421 INFO:teuthology.orchestra.run.smithi027.stdout:[osd] 2023-10-06T01:40:20.421 INFO:teuthology.orchestra.run.smithi027.stdout: osd journal size = 100 2023-10-06T01:40:20.421 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:40:20.421 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub load threshold = 5.0 2023-10-06T01:40:20.422 INFO:teuthology.orchestra.run.smithi027.stdout: osd scrub max interval = 600 2023-10-06T01:40:20.422 INFO:teuthology.orchestra.run.smithi027.stdout: osd mclock profile = high_recovery_ops 2023-10-06T01:40:20.422 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:40:20.422 INFO:teuthology.orchestra.run.smithi027.stdout: osd recover clone overlap = true 2023-10-06T01:40:20.422 INFO:teuthology.orchestra.run.smithi027.stdout: osd recovery max chunk = 1048576 2023-10-06T01:40:20.422 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:40:20.422 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug shutdown = true 2023-10-06T01:40:20.422 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug op order = true 2023-10-06T01:40:20.422 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug verify stray on activate = true 2023-10-06T01:40:20.422 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:40:20.422 INFO:teuthology.orchestra.run.smithi027.stdout: osd open classes on start = true 2023-10-06T01:40:20.422 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug pg log writeout = true 2023-10-06T01:40:20.423 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:40:20.423 INFO:teuthology.orchestra.run.smithi027.stdout: osd deep scrub update digest min age = 30 2023-10-06T01:40:20.423 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:40:20.423 INFO:teuthology.orchestra.run.smithi027.stdout: osd map max advance = 10 2023-10-06T01:40:20.423 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:40:20.423 INFO:teuthology.orchestra.run.smithi027.stdout: journal zero on create = true 2023-10-06T01:40:20.423 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:40:20.423 INFO:teuthology.orchestra.run.smithi027.stdout: filestore ondisk finisher threads = 3 2023-10-06T01:40:20.423 INFO:teuthology.orchestra.run.smithi027.stdout: filestore apply finisher threads = 3 2023-10-06T01:40:20.423 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:40:20.423 INFO:teuthology.orchestra.run.smithi027.stdout: bdev debug aio = true 2023-10-06T01:40:20.423 INFO:teuthology.orchestra.run.smithi027.stdout: osd debug misdirected ops = true 2023-10-06T01:40:20.424 INFO:teuthology.orchestra.run.smithi027.stdout: bdev async discard = True 2023-10-06T01:40:20.424 INFO:teuthology.orchestra.run.smithi027.stdout: bdev enable discard = True 2023-10-06T01:40:20.424 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore allocator = bitmap 2023-10-06T01:40:20.424 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore block size = 96636764160 2023-10-06T01:40:20.424 INFO:teuthology.orchestra.run.smithi027.stdout: bluestore fsck on mount = True 2023-10-06T01:40:20.424 INFO:teuthology.orchestra.run.smithi027.stdout: debug bluefs = 1/20 2023-10-06T01:40:20.424 INFO:teuthology.orchestra.run.smithi027.stdout: debug bluestore = 1/20 2023-10-06T01:40:20.424 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2023-10-06T01:40:20.424 INFO:teuthology.orchestra.run.smithi027.stdout: debug osd = 20 2023-10-06T01:40:20.424 INFO:teuthology.orchestra.run.smithi027.stdout: debug rocksdb = 4/10 2023-10-06T01:40:20.424 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd backfillfull_ratio = 0.85 2023-10-06T01:40:20.424 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd full ratio = 0.9 2023-10-06T01:40:20.424 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd nearfull ratio = 0.8 2023-10-06T01:40:20.425 INFO:teuthology.orchestra.run.smithi027.stdout: osd failsafe full ratio = 0.95 2023-10-06T01:40:20.425 INFO:teuthology.orchestra.run.smithi027.stdout: osd objectstore = bluestore 2023-10-06T01:40:20.425 INFO:teuthology.orchestra.run.smithi027.stdout: osd op complaint time = 180 2023-10-06T01:40:20.425 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:40:20.425 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr] 2023-10-06T01:40:20.425 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2023-10-06T01:40:20.425 INFO:teuthology.orchestra.run.smithi027.stdout: debug mgr = 20 2023-10-06T01:40:20.425 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2023-10-06T01:40:20.425 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2023-10-06T01:40:20.425 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2023-10-06T01:40:20.425 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2023-10-06T01:40:20.425 INFO:teuthology.orchestra.run.smithi027.stdout: mgr/telemetry/nag = false 2023-10-06T01:40:20.426 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:40:20.426 INFO:teuthology.orchestra.run.smithi027.stdout:[mon] 2023-10-06T01:40:20.426 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2023-10-06T01:40:20.426 INFO:teuthology.orchestra.run.smithi027.stdout: debug mon = 20 2023-10-06T01:40:20.426 INFO:teuthology.orchestra.run.smithi027.stdout: debug paxos = 20 2023-10-06T01:40:20.426 INFO:teuthology.orchestra.run.smithi027.stdout: debug auth = 20 2023-10-06T01:40:20.426 INFO:teuthology.orchestra.run.smithi027.stdout: mon data avail warn = 5 2023-10-06T01:40:20.426 INFO:teuthology.orchestra.run.smithi027.stdout: mon mgr mkfs grace = 240 2023-10-06T01:40:20.426 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min pgs per osd = 4 2023-10-06T01:40:20.426 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd reporter subtree level = osd 2023-10-06T01:40:20.426 INFO:teuthology.orchestra.run.smithi027.stdout: mon osd prime pg temp = true 2023-10-06T01:40:20.426 INFO:teuthology.orchestra.run.smithi027.stdout: mon reweight min bytes per osd = 10 2023-10-06T01:40:20.426 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:40:20.427 INFO:teuthology.orchestra.run.smithi027.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-06T01:40:20.427 INFO:teuthology.orchestra.run.smithi027.stdout: auth mon ticket ttl = 660 # 11m 2023-10-06T01:40:20.427 INFO:teuthology.orchestra.run.smithi027.stdout: auth service ticket ttl = 240 # 4m 2023-10-06T01:40:20.427 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:40:20.427 INFO:teuthology.orchestra.run.smithi027.stdout: # don't complain about insecure global_id in the test suite 2023-10-06T01:40:20.427 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-06T01:40:20.427 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-06T01:40:20.427 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:40:20.427 INFO:teuthology.orchestra.run.smithi027.stdout: # 1m isn't quite enough 2023-10-06T01:40:20.427 INFO:teuthology.orchestra.run.smithi027.stdout: mon_down_mkfs_grace = 2m 2023-10-06T01:40:20.427 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:40:20.427 INFO:teuthology.orchestra.run.smithi027.stdout: mon_warn_on_filestore_osds = false 2023-10-06T01:40:20.428 INFO:teuthology.orchestra.run.smithi027.stdout: mon op complaint time = 120 2023-10-06T01:40:20.428 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:40:20.428 INFO:teuthology.orchestra.run.smithi027.stdout:[client] 2023-10-06T01:40:20.428 INFO:teuthology.orchestra.run.smithi027.stdout: rgw cache enabled = true 2023-10-06T01:40:20.428 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable ops log = true 2023-10-06T01:40:20.428 INFO:teuthology.orchestra.run.smithi027.stdout: rgw enable usage log = true 2023-10-06T01:40:20.428 INFO:teuthology.orchestra.run.smithi027.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-06T01:40:20.428 INFO:teuthology.orchestra.run.smithi027.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-06T01:40:20.428 INFO:teuthology.orchestra.run.smithi027.stdout: client mount timeout = 600 2023-10-06T01:40:20.428 INFO:teuthology.orchestra.run.smithi027.stdout: debug client = 20 2023-10-06T01:40:20.428 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2023-10-06T01:40:20.428 INFO:teuthology.orchestra.run.smithi027.stdout: pid file = "" 2023-10-06T01:40:20.429 INFO:teuthology.orchestra.run.smithi027.stdout: rados mon op timeout = 900 2023-10-06T01:40:20.429 INFO:teuthology.orchestra.run.smithi027.stdout: rados osd op timeout = 900 2023-10-06T01:40:20.429 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.a] 2023-10-06T01:40:20.429 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.b] 2023-10-06T01:40:20.429 INFO:teuthology.orchestra.run.smithi027.stdout:[mon.c] 2023-10-06T01:40:20.429 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.a] 2023-10-06T01:40:20.429 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.c] 2023-10-06T01:40:20.429 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.b] 2023-10-06T01:40:20.429 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.d] 2023-10-06T01:40:20.429 INFO:teuthology.orchestra.run.smithi027.stdout:[mds] 2023-10-06T01:40:20.429 INFO:teuthology.orchestra.run.smithi027.stdout: debug mds = 20 2023-10-06T01:40:20.429 INFO:teuthology.orchestra.run.smithi027.stdout: debug ms = 1 2023-10-06T01:40:20.429 INFO:teuthology.orchestra.run.smithi027.stdout: mds debug frag = True 2023-10-06T01:40:20.430 INFO:teuthology.orchestra.run.smithi027.stdout: mds debug scatterstat = True 2023-10-06T01:40:20.430 INFO:teuthology.orchestra.run.smithi027.stdout: mds op complaint time = 180 2023-10-06T01:40:20.430 INFO:teuthology.orchestra.run.smithi027.stdout: mds verify scatter = True 2023-10-06T01:40:20.430 INFO:teuthology.orchestra.run.smithi027.stdout: osd op complaint time = 180 2023-10-06T01:40:20.430 INFO:teuthology.orchestra.run.smithi027.stdout: rados mon op timeout = 900 2023-10-06T01:40:20.430 INFO:teuthology.orchestra.run.smithi027.stdout: rados osd op timeout = 900 2023-10-06T01:40:20.431 INFO:teuthology.orchestra.run.smithi124.stdout:[global] 2023-10-06T01:40:20.431 INFO:teuthology.orchestra.run.smithi124.stdout: chdir = "" 2023-10-06T01:40:20.431 INFO:teuthology.orchestra.run.smithi124.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-06T01:40:20.432 INFO:teuthology.orchestra.run.smithi124.stdout: auth supported = cephx 2023-10-06T01:40:20.432 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-10-06T01:40:20.432 INFO:teuthology.orchestra.run.smithi124.stdout: filestore xattr use omap = true 2023-10-06T01:40:20.432 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-10-06T01:40:20.432 INFO:teuthology.orchestra.run.smithi124.stdout: mon clock drift allowed = 1.000 2023-10-06T01:40:20.432 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-10-06T01:40:20.432 INFO:teuthology.orchestra.run.smithi124.stdout: osd crush chooseleaf type = 0 2023-10-06T01:40:20.432 INFO:teuthology.orchestra.run.smithi124.stdout: auth debug = true 2023-10-06T01:40:20.432 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-10-06T01:40:20.432 INFO:teuthology.orchestra.run.smithi124.stdout: ms die on old message = true 2023-10-06T01:40:20.432 INFO:teuthology.orchestra.run.smithi124.stdout: ms die on bug = true 2023-10-06T01:40:20.432 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-10-06T01:40:20.433 INFO:teuthology.orchestra.run.smithi124.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-06T01:40:20.433 INFO:teuthology.orchestra.run.smithi124.stdout: mon pg warn max object skew = 0 2023-10-06T01:40:20.433 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-10-06T01:40:20.433 INFO:teuthology.orchestra.run.smithi124.stdout: # disable pg_autoscaler by default for new pools 2023-10-06T01:40:20.433 INFO:teuthology.orchestra.run.smithi124.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-06T01:40:20.433 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-10-06T01:40:20.433 INFO:teuthology.orchestra.run.smithi124.stdout: osd pool default size = 2 2023-10-06T01:40:20.433 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-10-06T01:40:20.433 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd allow primary affinity = true 2023-10-06T01:40:20.433 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd allow pg remap = true 2023-10-06T01:40:20.433 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on legacy crush tunables = false 2023-10-06T01:40:20.433 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on crush straw calc version zero = false 2023-10-06T01:40:20.433 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on no sortbitwise = false 2023-10-06T01:40:20.434 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on osd down out interval zero = false 2023-10-06T01:40:20.434 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on too few osds = false 2023-10-06T01:40:20.434 INFO:teuthology.orchestra.run.smithi124.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-06T01:40:20.434 INFO:teuthology.orchestra.run.smithi124.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-06T01:40:20.434 INFO:teuthology.orchestra.run.smithi124.stdout: mon_allow_pool_size_one = true 2023-10-06T01:40:20.434 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-10-06T01:40:20.434 INFO:teuthology.orchestra.run.smithi124.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-06T01:40:20.434 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-10-06T01:40:20.434 INFO:teuthology.orchestra.run.smithi124.stdout: osd default data pool replay window = 5 2023-10-06T01:40:20.434 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-10-06T01:40:20.434 INFO:teuthology.orchestra.run.smithi124.stdout: mon allow pool delete = true 2023-10-06T01:40:20.434 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-10-06T01:40:20.435 INFO:teuthology.orchestra.run.smithi124.stdout: mon cluster log file level = debug 2023-10-06T01:40:20.435 INFO:teuthology.orchestra.run.smithi124.stdout: debug asserts on shutdown = true 2023-10-06T01:40:20.435 INFO:teuthology.orchestra.run.smithi124.stdout: mon health detail to clog = false 2023-10-06T01:40:20.435 INFO:teuthology.orchestra.run.smithi124.stdout: mon host = "172.21.15.27,172.21.15.124,[v2:172.21.15.124:3301,v1:172.21.15.124:6790]" 2023-10-06T01:40:20.435 INFO:teuthology.orchestra.run.smithi124.stdout: ms die on skipped message = False 2023-10-06T01:40:20.435 INFO:teuthology.orchestra.run.smithi124.stdout: fsid = 20a7bff0-74f8-4f2f-b6c6-899e804bc965 2023-10-06T01:40:20.435 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-10-06T01:40:20.435 INFO:teuthology.orchestra.run.smithi124.stdout:[osd] 2023-10-06T01:40:20.435 INFO:teuthology.orchestra.run.smithi124.stdout: osd journal size = 100 2023-10-06T01:40:20.435 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-10-06T01:40:20.435 INFO:teuthology.orchestra.run.smithi124.stdout: osd scrub load threshold = 5.0 2023-10-06T01:40:20.435 INFO:teuthology.orchestra.run.smithi124.stdout: osd scrub max interval = 600 2023-10-06T01:40:20.436 INFO:teuthology.orchestra.run.smithi124.stdout: osd mclock profile = high_recovery_ops 2023-10-06T01:40:20.436 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-10-06T01:40:20.436 INFO:teuthology.orchestra.run.smithi124.stdout: osd recover clone overlap = true 2023-10-06T01:40:20.436 INFO:teuthology.orchestra.run.smithi124.stdout: osd recovery max chunk = 1048576 2023-10-06T01:40:20.436 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-10-06T01:40:20.436 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug shutdown = true 2023-10-06T01:40:20.436 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug op order = true 2023-10-06T01:40:20.436 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug verify stray on activate = true 2023-10-06T01:40:20.436 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-10-06T01:40:20.436 INFO:teuthology.orchestra.run.smithi124.stdout: osd open classes on start = true 2023-10-06T01:40:20.436 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug pg log writeout = true 2023-10-06T01:40:20.436 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-10-06T01:40:20.437 INFO:teuthology.orchestra.run.smithi124.stdout: osd deep scrub update digest min age = 30 2023-10-06T01:40:20.437 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-10-06T01:40:20.437 INFO:teuthology.orchestra.run.smithi124.stdout: osd map max advance = 10 2023-10-06T01:40:20.437 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-10-06T01:40:20.437 INFO:teuthology.orchestra.run.smithi124.stdout: journal zero on create = true 2023-10-06T01:40:20.437 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-10-06T01:40:20.437 INFO:teuthology.orchestra.run.smithi124.stdout: filestore ondisk finisher threads = 3 2023-10-06T01:40:20.437 INFO:teuthology.orchestra.run.smithi124.stdout: filestore apply finisher threads = 3 2023-10-06T01:40:20.437 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-10-06T01:40:20.437 INFO:teuthology.orchestra.run.smithi124.stdout: bdev debug aio = true 2023-10-06T01:40:20.437 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug misdirected ops = true 2023-10-06T01:40:20.437 INFO:teuthology.orchestra.run.smithi124.stdout: bdev async discard = True 2023-10-06T01:40:20.437 INFO:teuthology.orchestra.run.smithi124.stdout: bdev enable discard = True 2023-10-06T01:40:20.438 INFO:teuthology.orchestra.run.smithi124.stdout: bluestore allocator = bitmap 2023-10-06T01:40:20.438 INFO:teuthology.orchestra.run.smithi124.stdout: bluestore block size = 96636764160 2023-10-06T01:40:20.438 INFO:teuthology.orchestra.run.smithi124.stdout: bluestore fsck on mount = True 2023-10-06T01:40:20.438 INFO:teuthology.orchestra.run.smithi124.stdout: debug bluefs = 1/20 2023-10-06T01:40:20.438 INFO:teuthology.orchestra.run.smithi124.stdout: debug bluestore = 1/20 2023-10-06T01:40:20.438 INFO:teuthology.orchestra.run.smithi124.stdout: debug ms = 1 2023-10-06T01:40:20.438 INFO:teuthology.orchestra.run.smithi124.stdout: debug osd = 20 2023-10-06T01:40:20.438 INFO:teuthology.orchestra.run.smithi124.stdout: debug rocksdb = 4/10 2023-10-06T01:40:20.438 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd backfillfull_ratio = 0.85 2023-10-06T01:40:20.438 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd full ratio = 0.9 2023-10-06T01:40:20.438 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd nearfull ratio = 0.8 2023-10-06T01:40:20.438 INFO:teuthology.orchestra.run.smithi124.stdout: osd failsafe full ratio = 0.95 2023-10-06T01:40:20.439 INFO:teuthology.orchestra.run.smithi124.stdout: osd objectstore = bluestore 2023-10-06T01:40:20.439 INFO:teuthology.orchestra.run.smithi124.stdout: osd op complaint time = 180 2023-10-06T01:40:20.439 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-10-06T01:40:20.439 INFO:teuthology.orchestra.run.smithi124.stdout:[mgr] 2023-10-06T01:40:20.439 INFO:teuthology.orchestra.run.smithi124.stdout: debug ms = 1 2023-10-06T01:40:20.439 INFO:teuthology.orchestra.run.smithi124.stdout: debug mgr = 20 2023-10-06T01:40:20.439 INFO:teuthology.orchestra.run.smithi124.stdout: debug mon = 20 2023-10-06T01:40:20.439 INFO:teuthology.orchestra.run.smithi124.stdout: debug auth = 20 2023-10-06T01:40:20.439 INFO:teuthology.orchestra.run.smithi124.stdout: mon reweight min pgs per osd = 4 2023-10-06T01:40:20.439 INFO:teuthology.orchestra.run.smithi124.stdout: mon reweight min bytes per osd = 10 2023-10-06T01:40:20.439 INFO:teuthology.orchestra.run.smithi124.stdout: mgr/telemetry/nag = false 2023-10-06T01:40:20.439 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-10-06T01:40:20.439 INFO:teuthology.orchestra.run.smithi124.stdout:[mon] 2023-10-06T01:40:20.440 INFO:teuthology.orchestra.run.smithi124.stdout: debug ms = 1 2023-10-06T01:40:20.440 INFO:teuthology.orchestra.run.smithi124.stdout: debug mon = 20 2023-10-06T01:40:20.440 INFO:teuthology.orchestra.run.smithi124.stdout: debug paxos = 20 2023-10-06T01:40:20.440 INFO:teuthology.orchestra.run.smithi124.stdout: debug auth = 20 2023-10-06T01:40:20.440 INFO:teuthology.orchestra.run.smithi124.stdout: mon data avail warn = 5 2023-10-06T01:40:20.440 INFO:teuthology.orchestra.run.smithi124.stdout: mon mgr mkfs grace = 240 2023-10-06T01:40:20.440 INFO:teuthology.orchestra.run.smithi124.stdout: mon reweight min pgs per osd = 4 2023-10-06T01:40:20.440 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd reporter subtree level = osd 2023-10-06T01:40:20.440 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd prime pg temp = true 2023-10-06T01:40:20.440 INFO:teuthology.orchestra.run.smithi124.stdout: mon reweight min bytes per osd = 10 2023-10-06T01:40:20.440 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-10-06T01:40:20.440 INFO:teuthology.orchestra.run.smithi124.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-06T01:40:20.440 INFO:teuthology.orchestra.run.smithi124.stdout: auth mon ticket ttl = 660 # 11m 2023-10-06T01:40:20.441 INFO:teuthology.orchestra.run.smithi124.stdout: auth service ticket ttl = 240 # 4m 2023-10-06T01:40:20.441 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-10-06T01:40:20.441 INFO:teuthology.orchestra.run.smithi124.stdout: # don't complain about insecure global_id in the test suite 2023-10-06T01:40:20.441 INFO:teuthology.orchestra.run.smithi124.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-06T01:40:20.441 INFO:teuthology.orchestra.run.smithi124.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-06T01:40:20.441 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-10-06T01:40:20.441 INFO:teuthology.orchestra.run.smithi124.stdout: # 1m isn't quite enough 2023-10-06T01:40:20.441 INFO:teuthology.orchestra.run.smithi124.stdout: mon_down_mkfs_grace = 2m 2023-10-06T01:40:20.441 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-10-06T01:40:20.441 INFO:teuthology.orchestra.run.smithi124.stdout: mon_warn_on_filestore_osds = false 2023-10-06T01:40:20.441 INFO:teuthology.orchestra.run.smithi124.stdout: mon op complaint time = 120 2023-10-06T01:40:20.441 INFO:teuthology.orchestra.run.smithi124.stdout: 2023-10-06T01:40:20.441 INFO:teuthology.orchestra.run.smithi124.stdout:[client] 2023-10-06T01:40:20.442 INFO:teuthology.orchestra.run.smithi124.stdout: rgw cache enabled = true 2023-10-06T01:40:20.442 INFO:teuthology.orchestra.run.smithi124.stdout: rgw enable ops log = true 2023-10-06T01:40:20.442 INFO:teuthology.orchestra.run.smithi124.stdout: rgw enable usage log = true 2023-10-06T01:40:20.442 INFO:teuthology.orchestra.run.smithi124.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-06T01:40:20.442 INFO:teuthology.orchestra.run.smithi124.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-06T01:40:20.442 INFO:teuthology.orchestra.run.smithi124.stdout: client mount timeout = 600 2023-10-06T01:40:20.442 INFO:teuthology.orchestra.run.smithi124.stdout: debug client = 20 2023-10-06T01:40:20.442 INFO:teuthology.orchestra.run.smithi124.stdout: debug ms = 1 2023-10-06T01:40:20.442 INFO:teuthology.orchestra.run.smithi124.stdout: pid file = "" 2023-10-06T01:40:20.442 INFO:teuthology.orchestra.run.smithi124.stdout: rados mon op timeout = 900 2023-10-06T01:40:20.442 INFO:teuthology.orchestra.run.smithi124.stdout: rados osd op timeout = 900 2023-10-06T01:40:20.442 INFO:teuthology.orchestra.run.smithi124.stdout:[mon.a] 2023-10-06T01:40:20.442 INFO:teuthology.orchestra.run.smithi124.stdout:[mon.b] 2023-10-06T01:40:20.443 INFO:teuthology.orchestra.run.smithi124.stdout:[mon.c] 2023-10-06T01:40:20.443 INFO:teuthology.orchestra.run.smithi124.stdout:[mds.a] 2023-10-06T01:40:20.443 INFO:teuthology.orchestra.run.smithi124.stdout:[mds.c] 2023-10-06T01:40:20.443 INFO:teuthology.orchestra.run.smithi124.stdout:[mds.b] 2023-10-06T01:40:20.443 INFO:teuthology.orchestra.run.smithi124.stdout:[mds.d] 2023-10-06T01:40:20.443 INFO:teuthology.orchestra.run.smithi124.stdout:[mds] 2023-10-06T01:40:20.443 INFO:teuthology.orchestra.run.smithi124.stdout: debug mds = 20 2023-10-06T01:40:20.443 INFO:teuthology.orchestra.run.smithi124.stdout: debug ms = 1 2023-10-06T01:40:20.443 INFO:teuthology.orchestra.run.smithi124.stdout: mds debug frag = True 2023-10-06T01:40:20.443 INFO:teuthology.orchestra.run.smithi124.stdout: mds debug scatterstat = True 2023-10-06T01:40:20.443 INFO:teuthology.orchestra.run.smithi124.stdout: mds op complaint time = 180 2023-10-06T01:40:20.443 INFO:teuthology.orchestra.run.smithi124.stdout: mds verify scatter = True 2023-10-06T01:40:20.443 INFO:teuthology.orchestra.run.smithi124.stdout: osd op complaint time = 180 2023-10-06T01:40:20.444 INFO:teuthology.orchestra.run.smithi124.stdout: rados mon op timeout = 900 2023-10-06T01:40:20.444 INFO:teuthology.orchestra.run.smithi124.stdout: rados osd op timeout = 900 2023-10-06T01:40:20.444 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-06T01:40:20.444 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-06T01:40:20.523 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-06T01:40:20.524 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-06T01:40:20.524 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-06T01:40:20.539 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-06T01:40:20.539 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-06T01:40:20.593 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi027.front.sepia.ceph.com 2023-10-06T01:40:20.594 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-06T01:40:20.594 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-06T01:40:20.594 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-06T01:40:20.669 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-06T01:40:20.670 DEBUG:teuthology.orchestra.run.smithi027:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-06T01:40:20.724 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi124.front.sepia.ceph.com 2023-10-06T01:40:20.724 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-10-06T01:40:20.724 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-06T01:40:20.724 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-06T01:40:20.759 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-10-06T01:40:20.759 DEBUG:teuthology.orchestra.run.smithi124:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-06T01:40:20.813 INFO:tasks.ceph:Setting up mon nodes... 2023-10-06T01:40:20.814 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-06T01:40:20.814 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-10-06T01:40:20.863 INFO:teuthology.orchestra.run.smithi027.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-10-06T01:40:20.866 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-06T01:40:20.942 INFO:teuthology.orchestra.run.smithi124.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-06T01:40:20.945 INFO:tasks.ceph:Setting up mds nodes... 2023-10-06T01:40:20.945 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-10-06T01:40:20.994 INFO:teuthology.orchestra.run.smithi027.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-10-06T01:40:20.997 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-10-06T01:40:21.021 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-10-06T01:40:21.111 INFO:teuthology.orchestra.run.smithi027.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-10-06T01:40:21.113 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-10-06T01:40:21.138 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-10-06T01:40:21.185 INFO:teuthology.orchestra.run.smithi124.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-10-06T01:40:21.188 DEBUG:teuthology.orchestra.run.smithi124:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-10-06T01:40:21.212 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-10-06T01:40:21.299 INFO:teuthology.orchestra.run.smithi124.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-10-06T01:40:21.301 DEBUG:teuthology.orchestra.run.smithi124:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-10-06T01:40:21.325 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-06T01:40:21.325 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-06T01:40:21.364 INFO:teuthology.orchestra.run.smithi027.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-06T01:40:21.377 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2023-10-06T01:40:21.456 INFO:teuthology.orchestra.run.smithi027.stdout:creating /etc/ceph/ceph.client.2.keyring 2023-10-06T01:40:21.469 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-10-06T01:40:21.506 INFO:teuthology.orchestra.run.smithi124.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-10-06T01:40:21.518 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2023-10-06T01:40:21.596 INFO:teuthology.orchestra.run.smithi124.stdout:creating /etc/ceph/ceph.client.3.keyring 2023-10-06T01:40:21.608 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-06T01:40:21.609 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi027.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi124.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-06T01:40:21.609 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-06T01:40:21.634 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-06T01:40:21.634 INFO:tasks.ceph:role: osd.0 2023-10-06T01:40:21.634 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi027.front.sepia.ceph.com 2023-10-06T01:40:21.634 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-06T01:40:21.706 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-06T01:40:21.706 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-06T01:40:21.706 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-06T01:40:21.707 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 2023-10-06T01:40:21.707 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-06T01:40:21.707 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2023-10-06T01:40:21.707 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-06T01:40:21.707 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-06T01:40:21.707 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-06T01:40:21.707 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-06T01:40:22.131 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2023-10-06T01:40:22.133 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2023-10-06T01:40:22.133 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-06T01:40:22.302 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-06T01:40:22.334 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-06T01:40:22.400 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-06T01:40:22.400 INFO:tasks.ceph:role: osd.1 2023-10-06T01:40:22.400 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi027.front.sepia.ceph.com 2023-10-06T01:40:22.400 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-06T01:40:22.472 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-06T01:40:22.473 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-06T01:40:22.473 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-06T01:40:22.473 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 2023-10-06T01:40:22.473 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-06T01:40:22.473 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2023-10-06T01:40:22.473 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-06T01:40:22.473 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-06T01:40:22.473 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-06T01:40:22.473 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-06T01:40:22.898 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2023-10-06T01:40:22.900 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2023-10-06T01:40:22.900 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-06T01:40:23.012 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-06T01:40:23.042 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-06T01:40:23.107 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-06T01:40:23.108 INFO:tasks.ceph:role: osd.2 2023-10-06T01:40:23.108 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi027.front.sepia.ceph.com 2023-10-06T01:40:23.108 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-06T01:40:23.180 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-06T01:40:23.181 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-06T01:40:23.181 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-06T01:40:23.181 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 2023-10-06T01:40:23.181 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-06T01:40:23.181 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2023-10-06T01:40:23.181 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-06T01:40:23.181 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-06T01:40:23.181 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-06T01:40:23.181 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-06T01:40:23.626 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2023-10-06T01:40:23.628 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2023-10-06T01:40:23.628 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-06T01:40:23.798 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-06T01:40:23.830 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-10-06T01:40:23.895 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-06T01:40:23.895 INFO:tasks.ceph:role: osd.3 2023-10-06T01:40:23.895 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi027.front.sepia.ceph.com 2023-10-06T01:40:23.895 DEBUG:teuthology.orchestra.run.smithi027:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-06T01:40:23.967 INFO:teuthology.orchestra.run.smithi027.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-06T01:40:23.967 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-06T01:40:23.967 INFO:teuthology.orchestra.run.smithi027.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-06T01:40:23.967 INFO:teuthology.orchestra.run.smithi027.stdout: = reflink=1 2023-10-06T01:40:23.967 INFO:teuthology.orchestra.run.smithi027.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-06T01:40:23.967 INFO:teuthology.orchestra.run.smithi027.stdout: = sunit=0 swidth=0 blks 2023-10-06T01:40:23.967 INFO:teuthology.orchestra.run.smithi027.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-06T01:40:23.967 INFO:teuthology.orchestra.run.smithi027.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-06T01:40:23.968 INFO:teuthology.orchestra.run.smithi027.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-06T01:40:23.968 INFO:teuthology.orchestra.run.smithi027.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-06T01:40:24.401 INFO:teuthology.orchestra.run.smithi027.stdout:Discarding blocks...Done. 2023-10-06T01:40:24.402 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi027.front.sepia.ceph.com -o noatime 2023-10-06T01:40:24.402 DEBUG:teuthology.orchestra.run.smithi027:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-10-06T01:40:24.566 DEBUG:teuthology.orchestra.run.smithi027:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-10-06T01:40:24.600 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-06T01:40:24.935 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:24.934+0000 7f2863988540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-06T01:40:24.935 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:24.934+0000 7f2863988540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-06T01:40:24.936 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:24.934+0000 7f2863988540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-06T01:40:24.936 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:24.934+0000 7f2863988540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-06T01:40:24.936 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:24.934+0000 7f2863988540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-06T01:40:24.936 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:24.934+0000 7f2863988540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-06T01:40:25.467 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:25.466+0000 7f2863988540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T01:40:26.229 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:26.228+0000 7f2863988540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T01:40:26.992 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:26.991+0000 7f2863988540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T01:40:27.784 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-06T01:40:27.811 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-06T01:40:27.894 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:27.893+0000 7f9d194c1540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-06T01:40:27.895 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:27.893+0000 7f9d194c1540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-06T01:40:27.895 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:27.893+0000 7f9d194c1540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-06T01:40:27.895 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:27.893+0000 7f9d194c1540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-06T01:40:27.895 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:27.893+0000 7f9d194c1540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-06T01:40:27.895 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:27.894+0000 7f9d194c1540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-06T01:40:28.420 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:28.419+0000 7f9d194c1540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T01:40:29.185 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:29.184+0000 7f9d194c1540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T01:40:29.956 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:29.955+0000 7f9d194c1540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T01:40:30.754 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-06T01:40:30.825 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-06T01:40:30.929 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:30.928+0000 7f8f2706d540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-06T01:40:30.929 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:30.928+0000 7f8f2706d540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-06T01:40:30.929 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:30.929+0000 7f8f2706d540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-06T01:40:30.930 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:30.929+0000 7f8f2706d540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-06T01:40:30.930 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:30.929+0000 7f8f2706d540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-06T01:40:30.930 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:30.929+0000 7f8f2706d540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-06T01:40:31.465 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:31.465+0000 7f8f2706d540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T01:40:32.228 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:32.228+0000 7f8f2706d540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T01:40:32.990 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:32.990+0000 7f8f2706d540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T01:40:33.786 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-06T01:40:33.817 DEBUG:teuthology.orchestra.run.smithi027:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-06T01:40:33.904 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:33.904+0000 7feccdbaa540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-10-06T01:40:33.905 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:33.904+0000 7feccdbaa540 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-10-06T01:40:33.905 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:33.904+0000 7feccdbaa540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-06T01:40:33.905 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:33.904+0000 7feccdbaa540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-06T01:40:33.905 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:33.904+0000 7feccdbaa540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-06T01:40:33.905 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:33.905+0000 7feccdbaa540 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-10-06T01:40:34.435 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:34.435+0000 7feccdbaa540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T01:40:35.199 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:35.199+0000 7feccdbaa540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T01:40:35.966 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:35.966+0000 7feccdbaa540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T01:40:36.760 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-10-06T01:40:36.789 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-10-06T01:40:36.816 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-06T01:40:36.816 INFO:tasks.ceph:role: osd.4 2023-10-06T01:40:36.816 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi124.front.sepia.ceph.com 2023-10-06T01:40:36.817 DEBUG:teuthology.orchestra.run.smithi124:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-06T01:40:36.891 INFO:teuthology.orchestra.run.smithi124.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-06T01:40:36.891 INFO:teuthology.orchestra.run.smithi124.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-06T01:40:36.891 INFO:teuthology.orchestra.run.smithi124.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-06T01:40:36.891 INFO:teuthology.orchestra.run.smithi124.stdout: = reflink=1 2023-10-06T01:40:36.891 INFO:teuthology.orchestra.run.smithi124.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-06T01:40:36.891 INFO:teuthology.orchestra.run.smithi124.stdout: = sunit=0 swidth=0 blks 2023-10-06T01:40:36.891 INFO:teuthology.orchestra.run.smithi124.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-06T01:40:36.891 INFO:teuthology.orchestra.run.smithi124.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-06T01:40:36.892 INFO:teuthology.orchestra.run.smithi124.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-06T01:40:36.892 INFO:teuthology.orchestra.run.smithi124.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-06T01:40:37.318 INFO:teuthology.orchestra.run.smithi124.stdout:Discarding blocks...Done. 2023-10-06T01:40:37.319 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi124.front.sepia.ceph.com -o noatime 2023-10-06T01:40:37.320 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-10-06T01:40:37.479 DEBUG:teuthology.orchestra.run.smithi124:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-10-06T01:40:37.512 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-10-06T01:40:37.578 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-06T01:40:37.578 INFO:tasks.ceph:role: osd.5 2023-10-06T01:40:37.579 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi124.front.sepia.ceph.com 2023-10-06T01:40:37.579 DEBUG:teuthology.orchestra.run.smithi124:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-06T01:40:37.650 INFO:teuthology.orchestra.run.smithi124.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-06T01:40:37.650 INFO:teuthology.orchestra.run.smithi124.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-06T01:40:37.651 INFO:teuthology.orchestra.run.smithi124.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-06T01:40:37.651 INFO:teuthology.orchestra.run.smithi124.stdout: = reflink=1 2023-10-06T01:40:37.651 INFO:teuthology.orchestra.run.smithi124.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-06T01:40:37.651 INFO:teuthology.orchestra.run.smithi124.stdout: = sunit=0 swidth=0 blks 2023-10-06T01:40:37.651 INFO:teuthology.orchestra.run.smithi124.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-06T01:40:37.651 INFO:teuthology.orchestra.run.smithi124.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-06T01:40:37.651 INFO:teuthology.orchestra.run.smithi124.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-06T01:40:37.651 INFO:teuthology.orchestra.run.smithi124.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-06T01:40:38.092 INFO:teuthology.orchestra.run.smithi124.stdout:Discarding blocks...Done. 2023-10-06T01:40:38.093 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi124.front.sepia.ceph.com -o noatime 2023-10-06T01:40:38.093 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-10-06T01:40:38.194 DEBUG:teuthology.orchestra.run.smithi124:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-10-06T01:40:38.222 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-10-06T01:40:38.286 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-06T01:40:38.287 INFO:tasks.ceph:role: osd.6 2023-10-06T01:40:38.287 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi124.front.sepia.ceph.com 2023-10-06T01:40:38.287 DEBUG:teuthology.orchestra.run.smithi124:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-06T01:40:38.358 INFO:teuthology.orchestra.run.smithi124.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-06T01:40:38.359 INFO:teuthology.orchestra.run.smithi124.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-06T01:40:38.359 INFO:teuthology.orchestra.run.smithi124.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-06T01:40:38.359 INFO:teuthology.orchestra.run.smithi124.stdout: = reflink=1 2023-10-06T01:40:38.359 INFO:teuthology.orchestra.run.smithi124.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-06T01:40:38.359 INFO:teuthology.orchestra.run.smithi124.stdout: = sunit=0 swidth=0 blks 2023-10-06T01:40:38.359 INFO:teuthology.orchestra.run.smithi124.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-06T01:40:38.359 INFO:teuthology.orchestra.run.smithi124.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-06T01:40:38.359 INFO:teuthology.orchestra.run.smithi124.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-06T01:40:38.359 INFO:teuthology.orchestra.run.smithi124.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-06T01:40:38.806 INFO:teuthology.orchestra.run.smithi124.stdout:Discarding blocks...Done. 2023-10-06T01:40:38.808 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi124.front.sepia.ceph.com -o noatime 2023-10-06T01:40:38.808 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-10-06T01:40:38.938 DEBUG:teuthology.orchestra.run.smithi124:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-10-06T01:40:38.969 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-10-06T01:40:39.034 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-06T01:40:39.034 INFO:tasks.ceph:role: osd.7 2023-10-06T01:40:39.034 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi124.front.sepia.ceph.com 2023-10-06T01:40:39.035 DEBUG:teuthology.orchestra.run.smithi124:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-06T01:40:39.106 INFO:teuthology.orchestra.run.smithi124.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-06T01:40:39.106 INFO:teuthology.orchestra.run.smithi124.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-06T01:40:39.106 INFO:teuthology.orchestra.run.smithi124.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-06T01:40:39.106 INFO:teuthology.orchestra.run.smithi124.stdout: = reflink=1 2023-10-06T01:40:39.106 INFO:teuthology.orchestra.run.smithi124.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-06T01:40:39.106 INFO:teuthology.orchestra.run.smithi124.stdout: = sunit=0 swidth=0 blks 2023-10-06T01:40:39.106 INFO:teuthology.orchestra.run.smithi124.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-06T01:40:39.106 INFO:teuthology.orchestra.run.smithi124.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-06T01:40:39.106 INFO:teuthology.orchestra.run.smithi124.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-06T01:40:39.107 INFO:teuthology.orchestra.run.smithi124.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-06T01:40:39.547 INFO:teuthology.orchestra.run.smithi124.stdout:Discarding blocks...Done. 2023-10-06T01:40:39.548 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi124.front.sepia.ceph.com -o noatime 2023-10-06T01:40:39.549 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-10-06T01:40:39.714 DEBUG:teuthology.orchestra.run.smithi124:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-10-06T01:40:39.744 DEBUG:teuthology.orchestra.run.smithi124:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-06T01:40:39.838 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:39.836+0000 7f9debf83540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-10-06T01:40:39.838 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:39.836+0000 7f9debf83540 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-10-06T01:40:39.838 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:39.836+0000 7f9debf83540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-06T01:40:39.839 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:39.836+0000 7f9debf83540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-06T01:40:39.839 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:39.837+0000 7f9debf83540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-06T01:40:39.839 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:39.837+0000 7f9debf83540 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-10-06T01:40:40.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:40.361+0000 7f9debf83540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T01:40:41.128 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:41.127+0000 7f9debf83540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T01:40:41.890 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:41.888+0000 7f9debf83540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T01:40:42.681 DEBUG:teuthology.orchestra.run.smithi124:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-10-06T01:40:42.706 DEBUG:teuthology.orchestra.run.smithi124:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-06T01:40:42.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:42.785+0000 7fcbbba88540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-10-06T01:40:42.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:42.785+0000 7fcbbba88540 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-10-06T01:40:42.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:42.785+0000 7fcbbba88540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-06T01:40:42.786 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:42.785+0000 7fcbbba88540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-06T01:40:42.787 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:42.785+0000 7fcbbba88540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-06T01:40:42.787 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:42.785+0000 7fcbbba88540 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-10-06T01:40:43.308 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:43.307+0000 7fcbbba88540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T01:40:44.071 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:44.070+0000 7fcbbba88540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T01:40:44.837 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:44.836+0000 7fcbbba88540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T01:40:45.628 DEBUG:teuthology.orchestra.run.smithi124:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-10-06T01:40:45.695 DEBUG:teuthology.orchestra.run.smithi124:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-06T01:40:45.778 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:45.776+0000 7f50ce5dc540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-10-06T01:40:45.778 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:45.777+0000 7f50ce5dc540 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-10-06T01:40:45.779 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:45.777+0000 7f50ce5dc540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-06T01:40:45.779 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:45.777+0000 7f50ce5dc540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-06T01:40:45.779 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:45.777+0000 7f50ce5dc540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-06T01:40:45.779 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:45.777+0000 7f50ce5dc540 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-10-06T01:40:46.303 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:46.302+0000 7f50ce5dc540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T01:40:47.067 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:47.066+0000 7f50ce5dc540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T01:40:47.829 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:47.827+0000 7f50ce5dc540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T01:40:48.411 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T01:40:48.414 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T01:40:48.618 DEBUG:teuthology.orchestra.run.smithi124:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-10-06T01:40:48.643 DEBUG:teuthology.orchestra.run.smithi124:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-06T01:40:48.724 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:48.722+0000 7f4e7a6de540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-10-06T01:40:48.724 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:48.722+0000 7f4e7a6de540 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-10-06T01:40:48.724 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:48.722+0000 7f4e7a6de540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-06T01:40:48.724 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:48.722+0000 7f4e7a6de540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-06T01:40:48.724 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:48.723+0000 7f4e7a6de540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-06T01:40:48.725 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:48.723+0000 7f4e7a6de540 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-10-06T01:40:49.247 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:49.246+0000 7f4e7a6de540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T01:40:50.009 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:50.008+0000 7f4e7a6de540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T01:40:50.775 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:50.774+0000 7f4e7a6de540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-10-06T01:40:51.565 DEBUG:teuthology.orchestra.run.smithi124:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-10-06T01:40:51.631 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-06T01:40:51.631 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-06T01:40:51.632 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-10-06T01:40:51.659 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-06T01:40:51.659 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-10-06T01:40:51.726 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-06T01:40:51.726 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-10-06T01:40:51.792 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-06T01:40:51.793 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-06T01:40:51.859 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-06T01:40:51.859 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-06T01:40:51.925 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-06T01:40:51.925 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-06T01:40:51.990 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-06T01:40:51.990 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-10-06T01:40:52.056 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-10-06T01:40:52.056 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-06T01:40:52.083 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-10-06T01:40:52.083 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-10-06T01:40:52.148 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-10-06T01:40:52.149 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-10-06T01:40:52.213 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-10-06T01:40:52.213 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-10-06T01:40:52.277 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-10-06T01:40:52.277 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-10-06T01:40:52.341 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-10-06T01:40:52.341 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-10-06T01:40:52.405 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-10-06T01:40:52.405 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-10-06T01:40:52.468 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-06T01:40:52.468 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-06T01:40:52.484 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-06T01:40:52.484 DEBUG:teuthology.orchestra.run.smithi027:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2023-10-06T01:40:52.539 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-10-06T01:40:52.540 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-10-06T01:40:52.554 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-10-06T01:40:52.554 DEBUG:teuthology.orchestra.run.smithi124:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2023-10-06T01:40:52.608 INFO:tasks.ceph:Adding keys to all mons... 2023-10-06T01:40:52.608 DEBUG:teuthology.orchestra.run.smithi027:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-06T01:40:52.610 DEBUG:teuthology.orchestra.run.smithi124:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-06T01:40:52.651 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr.y] 2023-10-06T01:40:52.651 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCEZR9lV/5+MxAAaMkKkdnCzAPJAKk7KnEB0Q== 2023-10-06T01:40:52.651 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.a] 2023-10-06T01:40:52.652 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCEZR9lILtQOxAAQI7GMolNknD5BJTrG23HmA== 2023-10-06T01:40:52.652 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.c] 2023-10-06T01:40:52.652 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCFZR9lLf+qBhAAufgkGJq1ycJiPAd2W6w/Jg== 2023-10-06T01:40:52.652 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.0] 2023-10-06T01:40:52.652 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCIZR9lgBDDNxAArnRDoWNp3qAf8sQGLP62+g== 2023-10-06T01:40:52.652 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.1] 2023-10-06T01:40:52.652 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCLZR9l3rBTNRAArTUFBhj89Mt4WXoLD6Zu4w== 2023-10-06T01:40:52.652 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.2] 2023-10-06T01:40:52.652 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCOZR9lzz1iNxAAS9/IvCyrXwtoTEQAXbKIzw== 2023-10-06T01:40:52.652 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.3] 2023-10-06T01:40:52.652 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCRZR9lv2HsNRAA1jUJVb5QhD3r0i0r5NZFSA== 2023-10-06T01:40:52.653 INFO:teuthology.orchestra.run.smithi027.stdout:[mgr.x] 2023-10-06T01:40:52.653 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCEZR9l5QQ1OBAAdjOKHB5rfu3W34oUlB8Wmw== 2023-10-06T01:40:52.653 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.b] 2023-10-06T01:40:52.653 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCFZR9liOgYCxAAZ1OTCiARNIWws/bd7jyOUg== 2023-10-06T01:40:52.653 INFO:teuthology.orchestra.run.smithi027.stdout:[mds.d] 2023-10-06T01:40:52.653 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCFZR9lYybhERAAMmpeZMo/+odBszUSL8MTRQ== 2023-10-06T01:40:52.653 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.4] 2023-10-06T01:40:52.653 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCXZR9ljvX0MRAAFdyhJLaMeFcjE4s3nZu2dg== 2023-10-06T01:40:52.653 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.5] 2023-10-06T01:40:52.653 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCaZR9l9ErgLhAAF9++aR832vr6r/8K+MWHIA== 2023-10-06T01:40:52.653 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.6] 2023-10-06T01:40:52.653 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCdZR9l8EpkLhAATNtUt3vTRMVpi7kHG2gACA== 2023-10-06T01:40:52.654 INFO:teuthology.orchestra.run.smithi027.stdout:[osd.7] 2023-10-06T01:40:52.654 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCgZR9l0VspKxAAdD8MRmsSxkNE50yx+9RjZA== 2023-10-06T01:40:52.654 INFO:teuthology.orchestra.run.smithi027.stdout:[client.0] 2023-10-06T01:40:52.654 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCFZR9lizfBFRAAp7lQ8h3aUs5zyzqT0vz2Kw== 2023-10-06T01:40:52.654 INFO:teuthology.orchestra.run.smithi027.stdout:[client.2] 2023-10-06T01:40:52.654 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCFZR9lMwY1GxAAmB/1Xc/kVQZeDRJTW6Ca+w== 2023-10-06T01:40:52.654 INFO:teuthology.orchestra.run.smithi027.stdout:[client.1] 2023-10-06T01:40:52.654 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCFZR9lQAczHhAAxNf/t2BZLRUA1lbJqMLWDw== 2023-10-06T01:40:52.654 INFO:teuthology.orchestra.run.smithi027.stdout:[client.3] 2023-10-06T01:40:52.654 INFO:teuthology.orchestra.run.smithi027.stdout: key = AQCFZR9l7tOSIxAAxVuthAwmYrsU1HHI3ywf2g== 2023-10-06T01:40:52.670 INFO:teuthology.orchestra.run.smithi124.stdout:[mgr.y] 2023-10-06T01:40:52.670 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQCEZR9lV/5+MxAAaMkKkdnCzAPJAKk7KnEB0Q== 2023-10-06T01:40:52.670 INFO:teuthology.orchestra.run.smithi124.stdout:[mds.a] 2023-10-06T01:40:52.671 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQCEZR9lILtQOxAAQI7GMolNknD5BJTrG23HmA== 2023-10-06T01:40:52.671 INFO:teuthology.orchestra.run.smithi124.stdout:[mds.c] 2023-10-06T01:40:52.671 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQCFZR9lLf+qBhAAufgkGJq1ycJiPAd2W6w/Jg== 2023-10-06T01:40:52.671 INFO:teuthology.orchestra.run.smithi124.stdout:[osd.0] 2023-10-06T01:40:52.671 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQCIZR9lgBDDNxAArnRDoWNp3qAf8sQGLP62+g== 2023-10-06T01:40:52.671 INFO:teuthology.orchestra.run.smithi124.stdout:[osd.1] 2023-10-06T01:40:52.671 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQCLZR9l3rBTNRAArTUFBhj89Mt4WXoLD6Zu4w== 2023-10-06T01:40:52.671 INFO:teuthology.orchestra.run.smithi124.stdout:[osd.2] 2023-10-06T01:40:52.671 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQCOZR9lzz1iNxAAS9/IvCyrXwtoTEQAXbKIzw== 2023-10-06T01:40:52.671 INFO:teuthology.orchestra.run.smithi124.stdout:[osd.3] 2023-10-06T01:40:52.671 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQCRZR9lv2HsNRAA1jUJVb5QhD3r0i0r5NZFSA== 2023-10-06T01:40:52.671 INFO:teuthology.orchestra.run.smithi124.stdout:[mgr.x] 2023-10-06T01:40:52.672 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQCEZR9l5QQ1OBAAdjOKHB5rfu3W34oUlB8Wmw== 2023-10-06T01:40:52.672 INFO:teuthology.orchestra.run.smithi124.stdout:[mds.b] 2023-10-06T01:40:52.672 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQCFZR9liOgYCxAAZ1OTCiARNIWws/bd7jyOUg== 2023-10-06T01:40:52.672 INFO:teuthology.orchestra.run.smithi124.stdout:[mds.d] 2023-10-06T01:40:52.672 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQCFZR9lYybhERAAMmpeZMo/+odBszUSL8MTRQ== 2023-10-06T01:40:52.672 INFO:teuthology.orchestra.run.smithi124.stdout:[osd.4] 2023-10-06T01:40:52.672 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQCXZR9ljvX0MRAAFdyhJLaMeFcjE4s3nZu2dg== 2023-10-06T01:40:52.672 INFO:teuthology.orchestra.run.smithi124.stdout:[osd.5] 2023-10-06T01:40:52.672 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQCaZR9l9ErgLhAAF9++aR832vr6r/8K+MWHIA== 2023-10-06T01:40:52.672 INFO:teuthology.orchestra.run.smithi124.stdout:[osd.6] 2023-10-06T01:40:52.672 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQCdZR9l8EpkLhAATNtUt3vTRMVpi7kHG2gACA== 2023-10-06T01:40:52.672 INFO:teuthology.orchestra.run.smithi124.stdout:[osd.7] 2023-10-06T01:40:52.673 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQCgZR9l0VspKxAAdD8MRmsSxkNE50yx+9RjZA== 2023-10-06T01:40:52.673 INFO:teuthology.orchestra.run.smithi124.stdout:[client.0] 2023-10-06T01:40:52.673 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQCFZR9lizfBFRAAp7lQ8h3aUs5zyzqT0vz2Kw== 2023-10-06T01:40:52.673 INFO:teuthology.orchestra.run.smithi124.stdout:[client.2] 2023-10-06T01:40:52.673 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQCFZR9lMwY1GxAAmB/1Xc/kVQZeDRJTW6Ca+w== 2023-10-06T01:40:52.673 INFO:teuthology.orchestra.run.smithi124.stdout:[client.1] 2023-10-06T01:40:52.673 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQCFZR9lQAczHhAAxNf/t2BZLRUA1lbJqMLWDw== 2023-10-06T01:40:52.673 INFO:teuthology.orchestra.run.smithi124.stdout:[client.3] 2023-10-06T01:40:52.673 INFO:teuthology.orchestra.run.smithi124.stdout: key = AQCFZR9l7tOSIxAAxVuthAwmYrsU1HHI3ywf2g== 2023-10-06T01:40:52.674 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-06T01:40:52.697 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-06T01:40:52.760 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-06T01:40:52.763 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-06T01:40:52.813 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-06T01:40:52.816 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-06T01:40:52.866 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T01:40:52.868 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T01:40:52.918 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T01:40:52.920 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T01:40:52.967 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T01:40:52.970 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T01:40:53.017 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T01:40:53.019 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T01:40:53.067 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-06T01:40:53.070 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-06T01:40:53.117 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-06T01:40:53.119 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-06T01:40:53.169 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-06T01:40:53.171 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-06T01:40:53.221 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T01:40:53.223 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T01:40:53.270 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T01:40:53.272 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T01:40:53.319 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T01:40:53.321 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T01:40:53.368 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T01:40:53.370 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-06T01:40:53.417 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-06T01:40:53.419 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-06T01:40:53.469 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-06T01:40:53.471 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-06T01:40:53.520 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-06T01:40:53.522 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-06T01:40:53.572 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-06T01:40:53.574 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-06T01:40:53.624 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-06T01:40:53.624 DEBUG:teuthology.orchestra.run.smithi027:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-06T01:40:53.649 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-06T01:40:53.741 DEBUG:teuthology.orchestra.run.smithi027:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-06T01:40:53.766 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-10-06T01:40:53.791 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-06T01:40:53.882 DEBUG:teuthology.orchestra.run.smithi124:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-10-06T01:40:53.905 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-10-06T01:40:53.970 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-06T01:40:54.061 DEBUG:teuthology.orchestra.run.smithi124:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-10-06T01:40:54.085 DEBUG:teuthology.orchestra.run.smithi027:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-06T01:40:54.088 DEBUG:teuthology.orchestra.run.smithi124:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-06T01:40:54.141 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-06T01:40:54.141 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-06T01:40:54.141 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-06T01:40:54.143 INFO:tasks.ceph.mon.a:Started 2023-10-06T01:40:54.143 INFO:tasks.ceph.mon.b:Restarting daemon 2023-10-06T01:40:54.143 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-10-06T01:40:54.183 INFO:tasks.ceph.mon.b:Started 2023-10-06T01:40:54.183 INFO:tasks.ceph.mon.c:Restarting daemon 2023-10-06T01:40:54.183 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-10-06T01:40:54.185 INFO:tasks.ceph.mon.c:Started 2023-10-06T01:40:54.185 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-06T01:40:54.185 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-10-06T01:40:54.186 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-10-06T01:40:54.187 INFO:tasks.ceph.mgr.y:Started 2023-10-06T01:40:54.188 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-06T01:40:54.188 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-06T01:40:54.189 INFO:tasks.ceph.mgr.x:Started 2023-10-06T01:40:54.190 INFO:tasks.ceph:Setting crush tunables to default 2023-10-06T01:40:54.190 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph osd crush tunables default 2023-10-06T01:40:54.310 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.309+0000 7fbdf0fe0700 1 Processor -- start 2023-10-06T01:40:54.311 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.309+0000 7fbdf0fe0700 1 -- start start 2023-10-06T01:40:54.312 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.310+0000 7fbdf0fe0700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fbdec13fee0 0x7fbdec0c79c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:54.312 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.310+0000 7fbdf0fe0700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdec0c87a0 con 0x7fbdec06f270 2023-10-06T01:40:54.312 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.310+0000 7fbdf0fe0700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdec06adb0 con 0x7fbdec13fb10 2023-10-06T01:40:54.312 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.310+0000 7fbdf0fe0700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdec06b090 con 0x7fbdec13fee0 2023-10-06T01:40:54.312 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.310+0000 7fbdead9d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fbdec06f270 0x7fbdec0cb2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:36336/0 (socket says 172.21.15.27:36336) 2023-10-06T01:40:54.312 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.310+0000 7fbdead9d700 1 -- 172.21.15.27:0/3997935771 learned_addr learned my addr 172.21.15.27:0/3997935771 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:40:54.312 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.311+0000 7fbde9d9b700 1 --2- 172.21.15.27:0/3997935771 >> v2:172.21.15.124:3300/0 conn(0x7fbdec13fee0 0x7fbdec0c79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:54.313 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.311+0000 7fbde9d9b700 1 -- 172.21.15.27:0/3997935771 >> v2:172.21.15.124:3300/0 conn(0x7fbdec13fee0 msgr2=0x7fbdec0c79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-06T01:40:54.313 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.311+0000 7fbde9d9b700 1 -- 172.21.15.27:0/3997935771 >> v2:172.21.15.124:3300/0 conn(0x7fbdec13fee0 msgr2=0x7fbdec0c79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-06T01:40:54.313 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.311+0000 7fbde9d9b700 1 --2- 172.21.15.27:0/3997935771 >> v2:172.21.15.124:3300/0 conn(0x7fbdec13fee0 0x7fbdec0c79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-10-06T01:40:54.313 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.311+0000 7fbde9d9b700 1 --2- 172.21.15.27:0/3997935771 >> v2:172.21.15.124:3300/0 conn(0x7fbdec13fee0 0x7fbdec0c79c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-06T01:40:54.500 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.497+0000 7fbde959a700 1 -- 172.21.15.27:0/3997935771 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3090303813 0 0) 0x7fbdec0c87a0 con 0x7fbdec06f270 2023-10-06T01:40:54.500 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.497+0000 7fbde959a700 1 -- 172.21.15.27:0/3997935771 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdd8003570 con 0x7fbdec06f270 2023-10-06T01:40:54.500 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.497+0000 7fbde959a700 1 -- 172.21.15.27:0/3997935771 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2220620399 0 0) 0x7fbdec06adb0 con 0x7fbdec13fb10 2023-10-06T01:40:54.500 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.497+0000 7fbde959a700 1 -- 172.21.15.27:0/3997935771 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdec0c87a0 con 0x7fbdec13fb10 2023-10-06T01:40:54.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.497+0000 7fbde959a700 1 -- 172.21.15.27:0/3997935771 <== mon.1 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1809871188 0 0) 0x7fbdec0c87a0 con 0x7fbdec13fb10 2023-10-06T01:40:54.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.497+0000 7fbde959a700 1 -- 172.21.15.27:0/3997935771 >> v2:172.21.15.124:3300/0 conn(0x7fbdec13fee0 msgr2=0x7fbdec0c79c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-06T01:40:54.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.497+0000 7fbde959a700 1 --2- 172.21.15.27:0/3997935771 >> v2:172.21.15.124:3300/0 conn(0x7fbdec13fee0 0x7fbdec0c79c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:54.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.497+0000 7fbde959a700 1 -- 172.21.15.27:0/3997935771 >> v1:172.21.15.27:6789/0 conn(0x7fbdec06f270 legacy=0x7fbdec0cb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:54.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.497+0000 7fbde959a700 1 -- 172.21.15.27:0/3997935771 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdec06b370 con 0x7fbdec13fb10 2023-10-06T01:40:54.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.497+0000 7fbde959a700 1 -- 172.21.15.27:0/3997935771 <== mon.1 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2619078944 0 0) 0x7fbde4001470 con 0x7fbdec13fb10 2023-10-06T01:40:54.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.497+0000 7fbde959a700 1 -- 172.21.15.27:0/3997935771 >> v1:172.21.15.124:6789/0 conn(0x7fbdec13fb10 legacy=0x7fbdec06a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:54.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.497+0000 7fbde959a700 1 --2- 172.21.15.27:0/3997935771 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbdd8003bf0 0x7fbdd8003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:54.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.497+0000 7fbde959a700 1 --2- 172.21.15.27:0/3997935771 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbdd8004810 0x7fbdd8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:54.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.497+0000 7fbde959a700 1 --2- 172.21.15.27:0/3997935771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdd8005230 0x7fbdd800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:54.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.497+0000 7fbde959a700 1 -- 172.21.15.27:0/3997935771 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec06b090 con 0x7fbdd8005230 2023-10-06T01:40:54.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.497+0000 7fbde959a700 1 -- 172.21.15.27:0/3997935771 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd800c200 con 0x7fbdd8003bf0 2023-10-06T01:40:54.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.497+0000 7fbde959a700 1 -- 172.21.15.27:0/3997935771 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd800c3c0 con 0x7fbdd8004810 2023-10-06T01:40:54.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.497+0000 7fbdead9d700 1 --2- 172.21.15.27:0/3997935771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdd8005230 0x7fbdd800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:54.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.497+0000 7fbde9d9b700 1 --2- 172.21.15.27:0/3997935771 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbdd8004810 0x7fbdd8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:54.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.497+0000 7fbdea59c700 1 --2- 172.21.15.27:0/3997935771 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbdd8003bf0 0x7fbdd8003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:54.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.498+0000 7fbdead9d700 1 -- 172.21.15.27:0/3997935771 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbdd8004810 msgr2=0x7fbdd8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:40:54.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.498+0000 7fbdead9d700 1 --2- 172.21.15.27:0/3997935771 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbdd8004810 0x7fbdd8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:54.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.498+0000 7fbdead9d700 1 -- 172.21.15.27:0/3997935771 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbdd8003bf0 msgr2=0x7fbdd8003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:40:54.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.498+0000 7fbdead9d700 1 --2- 172.21.15.27:0/3997935771 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbdd8003bf0 0x7fbdd8003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:54.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.498+0000 7fbdead9d700 1 -- 172.21.15.27:0/3997935771 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdec0c87a0 con 0x7fbdd8005230 2023-10-06T01:40:54.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.498+0000 7fbdead9d700 1 -- 172.21.15.27:0/3997935771 reap_dead start 2023-10-06T01:40:54.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.498+0000 7fbdead9d700 1 --2- 172.21.15.27:0/3997935771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdd8005230 0x7fbdd800ba60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbddc00c440 tx=0x7fbddc001bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7996b0c4c205ace9 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:40:54.503 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.499+0000 7fbde959a700 1 -- 172.21.15.27:0/3997935771 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbddc001f00 con 0x7fbdd8005230 2023-10-06T01:40:54.503 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.499+0000 7fbde959a700 1 -- 172.21.15.27:0/3997935771 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbddc00cd70 con 0x7fbdd8005230 2023-10-06T01:40:54.503 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.499+0000 7fbde959a700 1 -- 172.21.15.27:0/3997935771 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbddc007300 con 0x7fbdd8005230 2023-10-06T01:40:54.504 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.500+0000 7fbdf0fe0700 1 -- 172.21.15.27:0/3997935771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdd8005230 msgr2=0x7fbdd800ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:54.505 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.500+0000 7fbdf0fe0700 1 --2- 172.21.15.27:0/3997935771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdd8005230 0x7fbdd800ba60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbddc00c440 tx=0x7fbddc001bd0 comp rx=0 tx=0).stop 2023-10-06T01:40:54.505 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.500+0000 7fbdf0fe0700 1 -- 172.21.15.27:0/3997935771 shutdown_connections 2023-10-06T01:40:54.505 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.500+0000 7fbdf0fe0700 1 --2- 172.21.15.27:0/3997935771 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdd8005230 0x7fbdd800ba60 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:54.505 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.500+0000 7fbdf0fe0700 1 -- 172.21.15.27:0/3997935771 >> 172.21.15.27:0/3997935771 conn(0x7fbdec06e760 msgr2=0x7fbdec06eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:40:54.505 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.500+0000 7fbdf0fe0700 1 -- 172.21.15.27:0/3997935771 shutdown_connections 2023-10-06T01:40:54.505 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.500+0000 7fbdf0fe0700 1 -- 172.21.15.27:0/3997935771 wait complete. 2023-10-06T01:40:54.505 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.501+0000 7fbdf0fe0700 1 Processor -- start 2023-10-06T01:40:54.505 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.501+0000 7fbdf0fe0700 1 -- start start 2023-10-06T01:40:54.505 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.502+0000 7fbdf0fe0700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdd8005230 0x7fbdec0b26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:54.505 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.502+0000 7fbdf0fe0700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbdec168a60 0x7fbdec16a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:54.505 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.502+0000 7fbdf0fe0700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbdec167840 0x7fbdec15af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.502+0000 7fbdf0fe0700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec16ac70 con 0x7fbdd8005230 2023-10-06T01:40:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.502+0000 7fbdf0fe0700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec168150 con 0x7fbdec167840 2023-10-06T01:40:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.502+0000 7fbdf0fe0700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdec168430 con 0x7fbdec168a60 2023-10-06T01:40:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.502+0000 7fbdea59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdd8005230 0x7fbdec0b26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.502+0000 7fbdea59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdd8005230 0x7fbdec0b26b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:58924/0 (socket says 172.21.15.27:58924) 2023-10-06T01:40:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.502+0000 7fbdea59c700 1 -- 172.21.15.27:0/2875657328 learned_addr learned my addr 172.21.15.27:0/2875657328 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:40:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.502+0000 7fbde9d9b700 1 --2- 172.21.15.27:0/2875657328 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbdec168a60 0x7fbdec16a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.502+0000 7fbdead9d700 1 --2- 172.21.15.27:0/2875657328 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbdec167840 0x7fbdec15af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.502+0000 7fbdea59c700 1 -- 172.21.15.27:0/2875657328 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbdec168a60 msgr2=0x7fbdec16a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.502+0000 7fbdea59c700 1 --2- 172.21.15.27:0/2875657328 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbdec168a60 0x7fbdec16a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.502+0000 7fbdea59c700 1 -- 172.21.15.27:0/2875657328 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbdec167840 msgr2=0x7fbdec15af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:54.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.502+0000 7fbdea59c700 1 --2- 172.21.15.27:0/2875657328 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbdec167840 0x7fbdec15af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:54.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.502+0000 7fbdea59c700 1 -- 172.21.15.27:0/2875657328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdec168710 con 0x7fbdd8005230 2023-10-06T01:40:54.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.503+0000 7fbdea59c700 1 --2- 172.21.15.27:0/2875657328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdd8005230 0x7fbdec0b26b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fbde4002bf0 tx=0x7fbdec06b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:40:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.504+0000 7fbde27fc700 1 -- 172.21.15.27:0/2875657328 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde4006a30 con 0x7fbdd8005230 2023-10-06T01:40:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.504+0000 7fbde27fc700 1 -- 172.21.15.27:0/2875657328 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde4006b90 con 0x7fbdd8005230 2023-10-06T01:40:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.504+0000 7fbdf0fe0700 1 -- 172.21.15.27:0/2875657328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdec0c4430 con 0x7fbdd8005230 2023-10-06T01:40:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.504+0000 7fbde27fc700 1 -- 172.21.15.27:0/2875657328 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde4007050 con 0x7fbdd8005230 2023-10-06T01:40:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.505+0000 7fbdf0fe0700 1 -- 172.21.15.27:0/2875657328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdec167ce0 con 0x7fbdd8005230 2023-10-06T01:40:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.505+0000 7fbde27fc700 1 -- 172.21.15.27:0/2875657328 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fbde400d5e0 con 0x7fbdd8005230 2023-10-06T01:40:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.505+0000 7fbde27fc700 1 -- 172.21.15.27:0/2875657328 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fbde401c750 con 0x7fbdd8005230 2023-10-06T01:40:54.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.505+0000 7fbdf0fe0700 1 -- 172.21.15.27:0/2875657328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdec0c4950 con 0x7fbdd8005230 2023-10-06T01:40:54.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.508+0000 7fbde27fc700 1 -- 172.21.15.27:0/2875657328 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fbde403b6d0 con 0x7fbdd8005230 2023-10-06T01:40:54.585 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.584+0000 7fbdf0fe0700 1 -- 172.21.15.27:0/2875657328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fbdec167f00 con 0x7fbdd8005230 2023-10-06T01:40:54.592 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.591+0000 7fbde27fc700 1 -- 172.21.15.27:0/2875657328 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fbde402a020 con 0x7fbdd8005230 2023-10-06T01:40:54.593 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.592+0000 7fbdf0fe0700 1 -- 172.21.15.27:0/2875657328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdd8005230 msgr2=0x7fbdec0b26b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:54.593 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.592+0000 7fbdf0fe0700 1 --2- 172.21.15.27:0/2875657328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdd8005230 0x7fbdec0b26b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fbde4002bf0 tx=0x7fbdec06b930 comp rx=0 tx=0).stop 2023-10-06T01:40:54.593 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.592+0000 7fbdf0fe0700 1 -- 172.21.15.27:0/2875657328 shutdown_connections 2023-10-06T01:40:54.593 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.592+0000 7fbdf0fe0700 1 --2- 172.21.15.27:0/2875657328 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbdec167840 0x7fbdec15af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:54.593 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.592+0000 7fbdf0fe0700 1 --2- 172.21.15.27:0/2875657328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdd8005230 0x7fbdec0b26b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:54.593 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.592+0000 7fbdf0fe0700 1 --2- 172.21.15.27:0/2875657328 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbdec168a60 0x7fbdec16a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:54.594 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.592+0000 7fbdf0fe0700 1 -- 172.21.15.27:0/2875657328 >> 172.21.15.27:0/2875657328 conn(0x7fbdec06e760 msgr2=0x7fbdec06eb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:40:54.594 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.592+0000 7fbdf0fe0700 1 -- 172.21.15.27:0/2875657328 shutdown_connections 2023-10-06T01:40:54.594 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:54.592+0000 7fbdf0fe0700 1 -- 172.21.15.27:0/2875657328 wait complete. 2023-10-06T01:40:54.594 INFO:teuthology.orchestra.run.smithi027.stderr:adjusted tunables profile to default 2023-10-06T01:40:54.605 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-06T01:40:54.605 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-06T01:40:54.606 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-06T01:40:54.630 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-06T01:40:54.630 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-06T01:40:54.695 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-06T01:40:54.695 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-06T01:40:54.761 DEBUG:teuthology.orchestra.run.smithi027:> set -ex 2023-10-06T01:40:54.762 DEBUG:teuthology.orchestra.run.smithi027:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-10-06T01:40:54.827 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-10-06T01:40:54.827 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-10-06T01:40:54.852 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-10-06T01:40:54.852 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-10-06T01:40:54.917 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-10-06T01:40:54.917 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-10-06T01:40:54.982 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2023-10-06T01:40:54.982 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-10-06T01:40:55.046 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --cluster ceph osd new b902c86d-800c-40c5-b152-b545e50d02c4 0 2023-10-06T01:40:55.202 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.200+0000 7f903e970700 1 Processor -- start 2023-10-06T01:40:55.202 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.200+0000 7f903e970700 1 -- start start 2023-10-06T01:40:55.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.201+0000 7f903e970700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9038159500 0x7f90381598d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:55.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.201+0000 7f903e970700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f90380b3fd0 0x7f903815d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:55.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.201+0000 7f903e970700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f903815dd20 con 0x7f903815d8e0 2023-10-06T01:40:55.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.201+0000 7f903e970700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90380b1110 con 0x7f9038159500 2023-10-06T01:40:55.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.201+0000 7f903e970700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90380b13b0 con 0x7f90380b3fd0 2023-10-06T01:40:55.205 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.201+0000 7f90377fe700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f90380b3fd0 0x7f903815d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:55.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.201+0000 7f903cf0d700 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f903815d8e0 0x7f90380cf6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:44058/0 (socket says 172.21.15.124:44058) 2023-10-06T01:40:55.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.201+0000 7f903cf0d700 1 -- 172.21.15.124:0/4123714307 learned_addr learned my addr 172.21.15.124:0/4123714307 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-10-06T01:40:55.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.202+0000 7f9037fff700 1 --2- 172.21.15.124:0/4123714307 >> v2:172.21.15.27:3300/0 conn(0x7f9038159500 0x7f90381598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:55.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.202+0000 7f9036ffd700 1 -- 172.21.15.124:0/4123714307 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2090223855 0 0) 0x7f903815dd20 con 0x7f903815d8e0 2023-10-06T01:40:55.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.202+0000 7f90377fe700 1 -- 172.21.15.124:0/4123714307 >> v2:172.21.15.27:3300/0 conn(0x7f9038159500 msgr2=0x7f90381598d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:40:55.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.202+0000 7f90377fe700 1 --2- 172.21.15.124:0/4123714307 >> v2:172.21.15.27:3300/0 conn(0x7f9038159500 0x7f90381598d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:55.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.202+0000 7f90377fe700 1 -- 172.21.15.124:0/4123714307 >> v1:172.21.15.124:6789/0 conn(0x7f903815d8e0 legacy=0x7f90380cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:55.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.202+0000 7f90377fe700 1 -- 172.21.15.124:0/4123714307 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90380b1690 con 0x7f90380b3fd0 2023-10-06T01:40:55.206 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.202+0000 7f90377fe700 1 --2- 172.21.15.124:0/4123714307 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f90380b3fd0 0x7f903815d3a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f90280098e0 tx=0x7f90280091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa8c185f9cfbfee3 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:40:55.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.202+0000 7f9036ffd700 1 -- 172.21.15.124:0/4123714307 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902800b640 con 0x7f90380b3fd0 2023-10-06T01:40:55.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.202+0000 7f9036ffd700 1 -- 172.21.15.124:0/4123714307 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f902800b7a0 con 0x7f90380b3fd0 2023-10-06T01:40:55.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.202+0000 7f9036ffd700 1 -- 172.21.15.124:0/4123714307 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f902800bb40 con 0x7f90380b3fd0 2023-10-06T01:40:55.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.203+0000 7f903e970700 1 -- 172.21.15.124:0/4123714307 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f90380b3fd0 msgr2=0x7f903815d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:55.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.203+0000 7f903e970700 1 --2- 172.21.15.124:0/4123714307 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f90380b3fd0 0x7f903815d3a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f90280098e0 tx=0x7f90280091a0 comp rx=0 tx=0).stop 2023-10-06T01:40:55.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.203+0000 7f903e970700 1 -- 172.21.15.124:0/4123714307 shutdown_connections 2023-10-06T01:40:55.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.203+0000 7f903e970700 1 --2- 172.21.15.124:0/4123714307 >> v2:172.21.15.27:3300/0 conn(0x7f9038159500 0x7f90381598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:55.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.203+0000 7f903e970700 1 --2- 172.21.15.124:0/4123714307 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f90380b3fd0 0x7f903815d3a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:55.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.203+0000 7f903e970700 1 -- 172.21.15.124:0/4123714307 >> 172.21.15.124:0/4123714307 conn(0x7f9038067630 msgr2=0x7f9038069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:40:55.207 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.203+0000 7f903e970700 1 -- 172.21.15.124:0/4123714307 shutdown_connections 2023-10-06T01:40:55.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.203+0000 7f903e970700 1 -- 172.21.15.124:0/4123714307 wait complete. 2023-10-06T01:40:55.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.204+0000 7f903e970700 1 Processor -- start 2023-10-06T01:40:55.208 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.204+0000 7f903e970700 1 -- start start 2023-10-06T01:40:55.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.205+0000 7f903e970700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f90380b3fd0 0x7f903814a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:55.210 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.205+0000 7f903e970700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9038159500 0x7f903816a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:55.211 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.205+0000 7f903e970700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f903815d8e0 0x7f9038154610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:55.211 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.205+0000 7f903e970700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9038153600 con 0x7f9038159500 2023-10-06T01:40:55.211 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.205+0000 7f903e970700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9038154180 con 0x7f90380b3fd0 2023-10-06T01:40:55.211 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.205+0000 7f903e970700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9038153850 con 0x7f903815d8e0 2023-10-06T01:40:55.211 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.205+0000 7f9037fff700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f90380b3fd0 0x7f903814a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:55.211 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.205+0000 7f903cf0d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f903815d8e0 0x7f9038154610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:55.211 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.205+0000 7f9037fff700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f90380b3fd0 0x7f903814a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:59240/0 (socket says 172.21.15.124:59240) 2023-10-06T01:40:55.211 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.205+0000 7f9037fff700 1 -- 172.21.15.124:0/1103246741 learned_addr learned my addr 172.21.15.124:0/1103246741 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-10-06T01:40:55.211 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.205+0000 7f90377fe700 1 --2- 172.21.15.124:0/1103246741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9038159500 0x7f903816a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:55.211 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.205+0000 7f903cf0d700 1 -- 172.21.15.124:0/1103246741 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f90380b3fd0 msgr2=0x7f903814a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:55.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.205+0000 7f903cf0d700 1 --2- 172.21.15.124:0/1103246741 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f90380b3fd0 0x7f903814a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:55.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.205+0000 7f903cf0d700 1 -- 172.21.15.124:0/1103246741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9038159500 msgr2=0x7f903816a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:55.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.205+0000 7f903cf0d700 1 --2- 172.21.15.124:0/1103246741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9038159500 0x7f903816a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:55.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.205+0000 7f903cf0d700 1 -- 172.21.15.124:0/1103246741 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9028009420 con 0x7f903815d8e0 2023-10-06T01:40:55.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.205+0000 7f9037fff700 1 --2- 172.21.15.124:0/1103246741 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f90380b3fd0 0x7f903814a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:40:55.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.205+0000 7f903cf0d700 1 --2- 172.21.15.124:0/1103246741 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f903815d8e0 0x7f9038154610 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f903000cbc0 tx=0x7f90300018f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:40:55.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.205+0000 7f901ffff700 1 -- 172.21.15.124:0/1103246741 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f903000a740 con 0x7f903815d8e0 2023-10-06T01:40:55.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.205+0000 7f901ffff700 1 -- 172.21.15.124:0/1103246741 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f903000a8c0 con 0x7f903815d8e0 2023-10-06T01:40:55.212 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.205+0000 7f903e970700 1 -- 172.21.15.124:0/1103246741 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9038153b30 con 0x7f903815d8e0 2023-10-06T01:40:55.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.206+0000 7f901ffff700 1 -- 172.21.15.124:0/1103246741 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f903000ac80 con 0x7f903815d8e0 2023-10-06T01:40:55.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.206+0000 7f903e970700 1 -- 172.21.15.124:0/1103246741 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f903814a8d0 con 0x7f903815d8e0 2023-10-06T01:40:55.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.206+0000 7f901ffff700 1 -- 172.21.15.124:0/1103246741 <== mon.2 v2:172.21.15.124:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f9030007300 con 0x7f903815d8e0 2023-10-06T01:40:55.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.206+0000 7f901ffff700 1 -- 172.21.15.124:0/1103246741 <== mon.2 v2:172.21.15.124:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7f903001b750 con 0x7f903815d8e0 2023-10-06T01:40:55.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.206+0000 7f903e970700 1 -- 172.21.15.124:0/1103246741 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f903814ab50 con 0x7f903815d8e0 2023-10-06T01:40:55.213 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.209+0000 7f901ffff700 1 -- 172.21.15.124:0/1103246741 <== mon.2 v2:172.21.15.124:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f903000ae30 con 0x7f903815d8e0 2023-10-06T01:40:55.292 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.291+0000 7f903e970700 1 -- 172.21.15.124:0/1103246741 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_command({"prefix": "osd new", "uuid": "b902c86d-800c-40c5-b152-b545e50d02c4", "id": 0} v 0) v1 -- 0x7f9038064050 con 0x7f903815d8e0 2023-10-06T01:40:55.298 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.297+0000 7f901ffff700 1 -- 172.21.15.124:0/1103246741 <== mon.2 v2:172.21.15.124:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b902c86d-800c-40c5-b152-b545e50d02c4", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9030007790 con 0x7f903815d8e0 2023-10-06T01:40:55.298 INFO:teuthology.orchestra.run.smithi124.stdout:0 2023-10-06T01:40:55.299 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.297+0000 7f903e970700 1 -- 172.21.15.124:0/1103246741 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f903815d8e0 msgr2=0x7f9038154610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:55.300 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.297+0000 7f903e970700 1 --2- 172.21.15.124:0/1103246741 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f903815d8e0 0x7f9038154610 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f903000cbc0 tx=0x7f90300018f0 comp rx=0 tx=0).stop 2023-10-06T01:40:55.300 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.297+0000 7f903e970700 1 -- 172.21.15.124:0/1103246741 shutdown_connections 2023-10-06T01:40:55.300 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.297+0000 7f903e970700 1 --2- 172.21.15.124:0/1103246741 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f903815d8e0 0x7f9038154610 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:55.300 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.298+0000 7f903e970700 1 --2- 172.21.15.124:0/1103246741 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f90380b3fd0 0x7f903814a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:55.300 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.298+0000 7f903e970700 1 --2- 172.21.15.124:0/1103246741 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9038159500 0x7f903816a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:55.300 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.298+0000 7f903e970700 1 -- 172.21.15.124:0/1103246741 >> 172.21.15.124:0/1103246741 conn(0x7f9038067630 msgr2=0x7f903814b370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:40:55.300 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.298+0000 7f903e970700 1 -- 172.21.15.124:0/1103246741 shutdown_connections 2023-10-06T01:40:55.300 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.298+0000 7f903e970700 1 -- 172.21.15.124:0/1103246741 wait complete. 2023-10-06T01:40:55.310 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --cluster ceph osd new 5b4a70e5-b06e-48cd-bfc4-7962c5d05386 1 2023-10-06T01:40:55.425 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.424+0000 7f095b59e700 1 Processor -- start 2023-10-06T01:40:55.426 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.424+0000 7f095b59e700 1 -- start start 2023-10-06T01:40:55.428 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.425+0000 7f095b59e700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f095c15de80 0x7f095c0b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:55.428 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.425+0000 7f095b59e700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f095c0b61d0 0x7f095c15d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:55.428 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.425+0000 7f095b59e700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f095c069380 con 0x7f095c142e90 2023-10-06T01:40:55.428 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.425+0000 7f095b59e700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f095c069f20 con 0x7f095c15de80 2023-10-06T01:40:55.428 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.425+0000 7f095b59e700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f095c068770 con 0x7f095c0b61d0 2023-10-06T01:40:55.429 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.425+0000 7f095ad9d700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f095c0b61d0 0x7f095c15d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:55.429 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.425+0000 7f095ad9d700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f095c0b61d0 0x7f095c15d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:59252/0 (socket says 172.21.15.124:59252) 2023-10-06T01:40:55.429 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.425+0000 7f095ad9d700 1 -- 172.21.15.124:0/2637000197 learned_addr learned my addr 172.21.15.124:0/2637000197 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-10-06T01:40:55.429 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.425+0000 7f095a59c700 1 --1- 172.21.15.124:0/2637000197 >> v1:172.21.15.124:6789/0 conn(0x7f095c142e90 0x7f095c069810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:44072/0 (socket says 172.21.15.124:44072) 2023-10-06T01:40:55.429 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.425+0000 7f095ad9d700 1 -- 172.21.15.124:0/2637000197 >> v2:172.21.15.27:3300/0 conn(0x7f095c15de80 msgr2=0x7f095c0b5c90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-06T01:40:55.429 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.425+0000 7f095ad9d700 1 --2- 172.21.15.124:0/2637000197 >> v2:172.21.15.27:3300/0 conn(0x7f095c15de80 0x7f095c0b5c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:55.429 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.425+0000 7f095ad9d700 1 -- 172.21.15.124:0/2637000197 >> v1:172.21.15.124:6789/0 conn(0x7f095c142e90 legacy=0x7f095c069810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:55.429 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.426+0000 7f095ad9d700 1 -- 172.21.15.124:0/2637000197 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f095c068a50 con 0x7f095c0b61d0 2023-10-06T01:40:55.429 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.425+0000 7f095959a700 1 -- 172.21.15.124:0/2637000197 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 409214367 0 0) 0x7f095c069380 con 0x7f095c142e90 2023-10-06T01:40:55.429 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.426+0000 7f095ad9d700 1 --2- 172.21.15.124:0/2637000197 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f095c0b61d0 0x7f095c15d290 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f095c0d0540 tx=0x7f0954009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d8d4ec91a2babc7 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:40:55.429 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.426+0000 7f095959a700 1 -- 172.21.15.124:0/2637000197 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f095400e070 con 0x7f095c0b61d0 2023-10-06T01:40:55.430 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.426+0000 7f095959a700 1 -- 172.21.15.124:0/2637000197 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f095400ba50 con 0x7f095c0b61d0 2023-10-06T01:40:55.430 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.426+0000 7f095959a700 1 -- 172.21.15.124:0/2637000197 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09540092e0 con 0x7f095c0b61d0 2023-10-06T01:40:55.430 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.426+0000 7f095b59e700 1 -- 172.21.15.124:0/2637000197 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f095c0b61d0 msgr2=0x7f095c15d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:55.430 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.426+0000 7f095b59e700 1 --2- 172.21.15.124:0/2637000197 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f095c0b61d0 0x7f095c15d290 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f095c0d0540 tx=0x7f0954009840 comp rx=0 tx=0).stop 2023-10-06T01:40:55.430 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.426+0000 7f095b59e700 1 -- 172.21.15.124:0/2637000197 shutdown_connections 2023-10-06T01:40:55.430 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.426+0000 7f095b59e700 1 --2- 172.21.15.124:0/2637000197 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f095c0b61d0 0x7f095c15d290 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:55.430 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.426+0000 7f095b59e700 1 --2- 172.21.15.124:0/2637000197 >> v2:172.21.15.27:3300/0 conn(0x7f095c15de80 0x7f095c0b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:55.430 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.426+0000 7f095b59e700 1 -- 172.21.15.124:0/2637000197 >> 172.21.15.124:0/2637000197 conn(0x7f095c0ccfe0 msgr2=0x7f095c0cd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:40:55.430 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.426+0000 7f095b59e700 1 -- 172.21.15.124:0/2637000197 shutdown_connections 2023-10-06T01:40:55.430 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.426+0000 7f095b59e700 1 -- 172.21.15.124:0/2637000197 wait complete. 2023-10-06T01:40:55.433 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.428+0000 7f095b59e700 1 Processor -- start 2023-10-06T01:40:55.433 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.428+0000 7f095b59e700 1 -- start start 2023-10-06T01:40:55.433 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.428+0000 7f095b59e700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f095c0b61d0 0x7f095c0b5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:55.433 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.428+0000 7f095b59e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f095c142e90 0x7f095c0c5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:55.433 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.428+0000 7f095b59e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f095c15de80 0x7f095c13e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:55.434 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.428+0000 7f095b59e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095c159440 con 0x7f095c15de80 2023-10-06T01:40:55.434 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.428+0000 7f095b59e700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095c0c59a0 con 0x7f095c0b61d0 2023-10-06T01:40:55.434 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.428+0000 7f095b59e700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f095c0c5c80 con 0x7f095c142e90 2023-10-06T01:40:55.434 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.428+0000 7f095a59c700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f095c0b61d0 0x7f095c0b5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:55.434 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.428+0000 7f095a59c700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f095c0b61d0 0x7f095c0b5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:59260/0 (socket says 172.21.15.124:59260) 2023-10-06T01:40:55.434 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.428+0000 7f095a59c700 1 -- 172.21.15.124:0/415138309 learned_addr learned my addr 172.21.15.124:0/415138309 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-10-06T01:40:55.434 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.428+0000 7f0959d9b700 1 --2- 172.21.15.124:0/415138309 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f095c142e90 0x7f095c0c5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:55.434 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.429+0000 7f095ad9d700 1 --2- 172.21.15.124:0/415138309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f095c15de80 0x7f095c13e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:55.435 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.429+0000 7f095a59c700 1 -- 172.21.15.124:0/415138309 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f095c142e90 msgr2=0x7f095c0c5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:55.435 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.429+0000 7f095a59c700 1 --2- 172.21.15.124:0/415138309 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f095c142e90 0x7f095c0c5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:55.435 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.429+0000 7f095a59c700 1 -- 172.21.15.124:0/415138309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f095c15de80 msgr2=0x7f095c13e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:55.435 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.429+0000 7f095a59c700 1 --2- 172.21.15.124:0/415138309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f095c15de80 0x7f095c13e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:55.435 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.429+0000 7f095a59c700 1 -- 172.21.15.124:0/415138309 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0954009ae0 con 0x7f095c0b61d0 2023-10-06T01:40:55.435 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.429+0000 7f0959d9b700 1 --2- 172.21.15.124:0/415138309 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f095c142e90 0x7f095c0c5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:40:55.435 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.429+0000 7f095a59c700 1 --2- 172.21.15.124:0/415138309 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f095c0b61d0 0x7f095c0b5c00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f09500011e0 tx=0x7f0950007e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:40:55.435 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.429+0000 7f094a7fc700 1 -- 172.21.15.124:0/415138309 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0950005a70 con 0x7f095c0b61d0 2023-10-06T01:40:55.435 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.429+0000 7f094a7fc700 1 -- 172.21.15.124:0/415138309 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0950005bd0 con 0x7f095c0b61d0 2023-10-06T01:40:55.435 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.429+0000 7f095b59e700 1 -- 172.21.15.124:0/415138309 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f095c0c5f60 con 0x7f095c0b61d0 2023-10-06T01:40:55.436 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.429+0000 7f094a7fc700 1 -- 172.21.15.124:0/415138309 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09500070d0 con 0x7f095c0b61d0 2023-10-06T01:40:55.436 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.429+0000 7f095b59e700 1 -- 172.21.15.124:0/415138309 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f095c0c6520 con 0x7f095c0b61d0 2023-10-06T01:40:55.436 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.429+0000 7f094a7fc700 1 -- 172.21.15.124:0/415138309 <== mon.1 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f095001c440 con 0x7f095c0b61d0 2023-10-06T01:40:55.436 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.429+0000 7f094a7fc700 1 -- 172.21.15.124:0/415138309 <== mon.1 v2:172.21.15.124:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7f09500127c0 con 0x7f095c0b61d0 2023-10-06T01:40:55.436 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.430+0000 7f095b59e700 1 -- 172.21.15.124:0/415138309 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f095c0c6740 con 0x7f095c0b61d0 2023-10-06T01:40:55.436 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.432+0000 7f094a7fc700 1 -- 172.21.15.124:0/415138309 <== mon.1 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f0950017070 con 0x7f095c0b61d0 2023-10-06T01:40:55.517 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.516+0000 7f095b59e700 1 -- 172.21.15.124:0/415138309 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5b4a70e5-b06e-48cd-bfc4-7962c5d05386", "id": 1} v 0) v1 -- 0x7f095c04c3c0 con 0x7f095c0b61d0 2023-10-06T01:40:55.524 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.522+0000 7f094a7fc700 1 -- 172.21.15.124:0/415138309 <== mon.1 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5b4a70e5-b06e-48cd-bfc4-7962c5d05386", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f095001c8d0 con 0x7f095c0b61d0 2023-10-06T01:40:55.524 INFO:teuthology.orchestra.run.smithi124.stdout:1 2023-10-06T01:40:55.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.523+0000 7f095b59e700 1 -- 172.21.15.124:0/415138309 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f095c0b61d0 msgr2=0x7f095c0b5c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:55.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.523+0000 7f095b59e700 1 --2- 172.21.15.124:0/415138309 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f095c0b61d0 0x7f095c0b5c00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f09500011e0 tx=0x7f0950007e10 comp rx=0 tx=0).stop 2023-10-06T01:40:55.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.523+0000 7f095b59e700 1 -- 172.21.15.124:0/415138309 shutdown_connections 2023-10-06T01:40:55.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.523+0000 7f095b59e700 1 --2- 172.21.15.124:0/415138309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f095c15de80 0x7f095c13e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:55.525 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.523+0000 7f095b59e700 1 --2- 172.21.15.124:0/415138309 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f095c0b61d0 0x7f095c0b5c00 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:55.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.523+0000 7f095b59e700 1 --2- 172.21.15.124:0/415138309 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f095c142e90 0x7f095c0c5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:55.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.523+0000 7f095b59e700 1 -- 172.21.15.124:0/415138309 >> 172.21.15.124:0/415138309 conn(0x7f095c0ccfe0 msgr2=0x7f095c1407c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:40:55.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.523+0000 7f095b59e700 1 -- 172.21.15.124:0/415138309 shutdown_connections 2023-10-06T01:40:55.526 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.523+0000 7f095b59e700 1 -- 172.21.15.124:0/415138309 wait complete. 2023-10-06T01:40:55.536 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --cluster ceph osd new 22d9de46-4073-4336-9122-5227e601eca0 2 2023-10-06T01:40:55.651 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.650+0000 7f3d6fa11700 1 Processor -- start 2023-10-06T01:40:55.652 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.650+0000 7f3d6fa11700 1 -- start start 2023-10-06T01:40:55.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.651+0000 7f3d6fa11700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3d68157b50 0x7f3d680cd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:55.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.651+0000 7f3d6fa11700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3d680cdb30 0x7f3d68157f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:55.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.651+0000 7f3d6fa11700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d68159a60 con 0x7f3d68158460 2023-10-06T01:40:55.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.651+0000 7f3d6fa11700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d68158e50 con 0x7f3d68157b50 2023-10-06T01:40:55.654 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.651+0000 7f3d6fa11700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d68159130 con 0x7f3d680cdb30 2023-10-06T01:40:55.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.651+0000 7f3d6cfac700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3d680cdb30 0x7f3d68157f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:55.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.651+0000 7f3d6cfac700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3d680cdb30 0x7f3d68157f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.124:43358/0 (socket says 172.21.15.124:43358) 2023-10-06T01:40:55.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.651+0000 7f3d6cfac700 1 -- 172.21.15.124:0/3296276665 learned_addr learned my addr 172.21.15.124:0/3296276665 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-10-06T01:40:55.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.652+0000 7f3d6d7ad700 1 --2- 172.21.15.124:0/3296276665 >> v2:172.21.15.27:3300/0 conn(0x7f3d68157b50 0x7f3d680cd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:55.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.652+0000 7f3d6cfac700 1 -- 172.21.15.124:0/3296276665 >> v2:172.21.15.27:3300/0 conn(0x7f3d68157b50 msgr2=0x7f3d680cd5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:40:55.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.652+0000 7f3d6cfac700 1 --2- 172.21.15.124:0/3296276665 >> v2:172.21.15.27:3300/0 conn(0x7f3d68157b50 0x7f3d680cd5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:55.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.652+0000 7f3d6cfac700 1 -- 172.21.15.124:0/3296276665 >> v1:172.21.15.27:6789/0 conn(0x7f3d68158460 legacy=0x7f3d68159ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:55.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.652+0000 7f3d6cfac700 1 -- 172.21.15.124:0/3296276665 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d68159410 con 0x7f3d680cdb30 2023-10-06T01:40:55.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.652+0000 7f3d6cfac700 1 --2- 172.21.15.124:0/3296276665 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3d680cdb30 0x7f3d68157f20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3d640099b0 tx=0x7f3d64009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b79d40e88bb77ff server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:40:55.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.652+0000 7f3d5ffff700 1 -- 172.21.15.124:0/3296276665 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d64009ea0 con 0x7f3d680cdb30 2023-10-06T01:40:55.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.652+0000 7f3d5ffff700 1 -- 172.21.15.124:0/3296276665 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d6400b860 con 0x7f3d680cdb30 2023-10-06T01:40:55.655 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.652+0000 7f3d5ffff700 1 -- 172.21.15.124:0/3296276665 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6400bc00 con 0x7f3d680cdb30 2023-10-06T01:40:55.656 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.653+0000 7f3d6fa11700 1 -- 172.21.15.124:0/3296276665 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3d680cdb30 msgr2=0x7f3d68157f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:55.656 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.653+0000 7f3d6fa11700 1 --2- 172.21.15.124:0/3296276665 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3d680cdb30 0x7f3d68157f20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3d640099b0 tx=0x7f3d64009250 comp rx=0 tx=0).stop 2023-10-06T01:40:55.656 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.653+0000 7f3d6fa11700 1 -- 172.21.15.124:0/3296276665 shutdown_connections 2023-10-06T01:40:55.656 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.653+0000 7f3d6fa11700 1 --2- 172.21.15.124:0/3296276665 >> v2:172.21.15.27:3300/0 conn(0x7f3d68157b50 0x7f3d680cd5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:55.656 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.653+0000 7f3d6fa11700 1 --2- 172.21.15.124:0/3296276665 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3d680cdb30 0x7f3d68157f20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:55.656 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.653+0000 7f3d6fa11700 1 -- 172.21.15.124:0/3296276665 >> 172.21.15.124:0/3296276665 conn(0x7f3d6806d890 msgr2=0x7f3d6806dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:40:55.656 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.653+0000 7f3d6fa11700 1 -- 172.21.15.124:0/3296276665 shutdown_connections 2023-10-06T01:40:55.656 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.653+0000 7f3d6fa11700 1 -- 172.21.15.124:0/3296276665 wait complete. 2023-10-06T01:40:55.658 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.654+0000 7f3d6fa11700 1 Processor -- start 2023-10-06T01:40:55.658 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.654+0000 7f3d6fa11700 1 -- start start 2023-10-06T01:40:55.658 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.655+0000 7f3d6fa11700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3d680cdb30 0x7f3d6813b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:55.659 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.655+0000 7f3d6fa11700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d68157b50 0x7f3d6815be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:55.659 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.655+0000 7f3d6fa11700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3d68158460 0x7f3d6815c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:55.659 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.655+0000 7f3d6fa11700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6816c070 con 0x7f3d68157b50 2023-10-06T01:40:55.659 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.655+0000 7f3d6fa11700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6813bed0 con 0x7f3d68158460 2023-10-06T01:40:55.659 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.655+0000 7f3d6fa11700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d68153d30 con 0x7f3d680cdb30 2023-10-06T01:40:55.659 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.655+0000 7f3d6d7ad700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3d680cdb30 0x7f3d6813b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:55.659 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.655+0000 7f3d6d7ad700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3d680cdb30 0x7f3d6813b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.124:43362/0 (socket says 172.21.15.124:43362) 2023-10-06T01:40:55.659 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.655+0000 7f3d6d7ad700 1 -- 172.21.15.124:0/4208953449 learned_addr learned my addr 172.21.15.124:0/4208953449 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-10-06T01:40:55.659 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.655+0000 7f3d6dfae700 1 --2- 172.21.15.124:0/4208953449 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3d68158460 0x7f3d6815c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:55.659 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.655+0000 7f3d6cfac700 1 --2- 172.21.15.124:0/4208953449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d68157b50 0x7f3d6815be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:55.660 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.655+0000 7f3d6d7ad700 1 -- 172.21.15.124:0/4208953449 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3d68158460 msgr2=0x7f3d6815c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:55.660 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.655+0000 7f3d6d7ad700 1 --2- 172.21.15.124:0/4208953449 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3d68158460 0x7f3d6815c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:55.660 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.655+0000 7f3d6d7ad700 1 -- 172.21.15.124:0/4208953449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d68157b50 msgr2=0x7f3d6815be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:55.660 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.655+0000 7f3d6d7ad700 1 --2- 172.21.15.124:0/4208953449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d68157b50 0x7f3d6815be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:55.660 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.655+0000 7f3d6d7ad700 1 -- 172.21.15.124:0/4208953449 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d640094f0 con 0x7f3d680cdb30 2023-10-06T01:40:55.660 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.655+0000 7f3d6d7ad700 1 --2- 172.21.15.124:0/4208953449 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3d680cdb30 0x7f3d6813b990 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3d5800ecb0 tx=0x7f3d580091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:40:55.660 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.655+0000 7f3d5d7fa700 1 -- 172.21.15.124:0/4208953449 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d5800c850 con 0x7f3d680cdb30 2023-10-06T01:40:55.660 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.655+0000 7f3d5d7fa700 1 -- 172.21.15.124:0/4208953449 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d5800c9b0 con 0x7f3d680cdb30 2023-10-06T01:40:55.660 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.655+0000 7f3d5d7fa700 1 -- 172.21.15.124:0/4208953449 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d5800cd50 con 0x7f3d680cdb30 2023-10-06T01:40:55.660 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.655+0000 7f3d6fa11700 1 -- 172.21.15.124:0/4208953449 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d68154010 con 0x7f3d680cdb30 2023-10-06T01:40:55.660 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.655+0000 7f3d6fa11700 1 -- 172.21.15.124:0/4208953449 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d681545d0 con 0x7f3d680cdb30 2023-10-06T01:40:55.660 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.656+0000 7f3d5d7fa700 1 -- 172.21.15.124:0/4208953449 <== mon.2 v2:172.21.15.124:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f3d58005c80 con 0x7f3d680cdb30 2023-10-06T01:40:55.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.656+0000 7f3d5d7fa700 1 -- 172.21.15.124:0/4208953449 <== mon.2 v2:172.21.15.124:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7f3d58006020 con 0x7f3d680cdb30 2023-10-06T01:40:55.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.657+0000 7f3d6fa11700 1 -- 172.21.15.124:0/4208953449 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d68064050 con 0x7f3d680cdb30 2023-10-06T01:40:55.661 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.659+0000 7f3d5d7fa700 1 -- 172.21.15.124:0/4208953449 <== mon.2 v2:172.21.15.124:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f3d58021060 con 0x7f3d680cdb30 2023-10-06T01:40:55.692 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-10-06T01:40:55.691+0000 7fc8049c8180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-06T01:40:55.692 INFO:tasks.ceph.mgr.x.smithi124.stderr:2023-10-06T01:40:55.692+0000 7fdf65db9180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-06T01:40:55.775 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.774+0000 7f3d6fa11700 1 -- 172.21.15.124:0/4208953449 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_command({"prefix": "osd new", "uuid": "22d9de46-4073-4336-9122-5227e601eca0", "id": 2} v 0) v1 -- 0x7f3d680cf290 con 0x7f3d680cdb30 2023-10-06T01:40:55.779 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.778+0000 7f3d5d7fa700 1 -- 172.21.15.124:0/4208953449 <== mon.2 v2:172.21.15.124:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "22d9de46-4073-4336-9122-5227e601eca0", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3d58017070 con 0x7f3d680cdb30 2023-10-06T01:40:55.779 INFO:teuthology.orchestra.run.smithi124.stdout:2 2023-10-06T01:40:55.781 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.779+0000 7f3d6fa11700 1 -- 172.21.15.124:0/4208953449 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3d680cdb30 msgr2=0x7f3d6813b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:55.781 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.779+0000 7f3d6fa11700 1 --2- 172.21.15.124:0/4208953449 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3d680cdb30 0x7f3d6813b990 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3d5800ecb0 tx=0x7f3d580091a0 comp rx=0 tx=0).stop 2023-10-06T01:40:55.781 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.779+0000 7f3d6fa11700 1 -- 172.21.15.124:0/4208953449 shutdown_connections 2023-10-06T01:40:55.781 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.779+0000 7f3d6fa11700 1 --2- 172.21.15.124:0/4208953449 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3d680cdb30 0x7f3d6813b990 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:55.781 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.779+0000 7f3d6fa11700 1 --2- 172.21.15.124:0/4208953449 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3d68158460 0x7f3d6815c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:55.781 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.779+0000 7f3d6fa11700 1 --2- 172.21.15.124:0/4208953449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d68157b50 0x7f3d6815be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:55.781 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.779+0000 7f3d6fa11700 1 -- 172.21.15.124:0/4208953449 >> 172.21.15.124:0/4208953449 conn(0x7f3d6806d890 msgr2=0x7f3d680b3750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:40:55.781 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.779+0000 7f3d6fa11700 1 -- 172.21.15.124:0/4208953449 shutdown_connections 2023-10-06T01:40:55.781 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.779+0000 7f3d6fa11700 1 -- 172.21.15.124:0/4208953449 wait complete. 2023-10-06T01:40:55.796 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --cluster ceph osd new 438bb2d3-34e2-4b7c-9734-8360c1b67650 3 2023-10-06T01:40:55.828 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-10-06T01:40:55.827+0000 7fc8049c8180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-06T01:40:55.847 INFO:tasks.ceph.mgr.x.smithi124.stderr:2023-10-06T01:40:55.846+0000 7fdf65db9180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-06T01:40:55.906 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-10-06T01:40:55.905+0000 7fc8049c8180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-06T01:40:55.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.910+0000 7f723dadf700 1 Processor -- start 2023-10-06T01:40:55.911 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.910+0000 7f723dadf700 1 -- start start 2023-10-06T01:40:55.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.910+0000 7f723dadf700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f72380b4000 0x7f7238159b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:55.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.910+0000 7f723dadf700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72380698e0 con 0x7f72380b1690 2023-10-06T01:40:55.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.910+0000 7f723dadf700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7238068fd0 con 0x7f723815cb10 2023-10-06T01:40:55.913 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.910+0000 7f723dadf700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72380b0020 con 0x7f72380b4000 2023-10-06T01:40:55.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.911+0000 7f72377fe700 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f723815cb10 0x7f72380af780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:44082/0 (socket says 172.21.15.124:44082) 2023-10-06T01:40:55.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.911+0000 7f72377fe700 1 -- 172.21.15.124:0/2076717850 learned_addr learned my addr 172.21.15.124:0/2076717850 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-10-06T01:40:55.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.911+0000 7f722ffff700 1 --2- 172.21.15.124:0/2076717850 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f72380b4000 0x7f7238159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:55.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.911+0000 7f7236ffd700 1 -- 172.21.15.124:0/2076717850 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2803813547 0 0) 0x7f7238068fd0 con 0x7f723815cb10 2023-10-06T01:40:55.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.911+0000 7f7236ffd700 1 -- 172.21.15.124:0/2076717850 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7224003570 con 0x7f723815cb10 2023-10-06T01:40:55.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.911+0000 7f722ffff700 1 -- 172.21.15.124:0/2076717850 >> v1:172.21.15.124:6789/0 conn(0x7f723815cb10 legacy=0x7f72380af780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:55.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.911+0000 7f722ffff700 1 -- 172.21.15.124:0/2076717850 >> v1:172.21.15.27:6789/0 conn(0x7f72380b1690 legacy=0x7f7238069d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:55.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.911+0000 7f722ffff700 1 -- 172.21.15.124:0/2076717850 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7238069290 con 0x7f72380b4000 2023-10-06T01:40:55.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.911+0000 7f722ffff700 1 --2- 172.21.15.124:0/2076717850 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f72380b4000 0x7f7238159b50 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7220009910 tx=0x7f72200091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc96abb91d71655a server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:40:55.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.911+0000 7f7236ffd700 1 -- 172.21.15.124:0/2076717850 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722000b680 con 0x7f72380b4000 2023-10-06T01:40:55.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.911+0000 7f7236ffd700 1 -- 172.21.15.124:0/2076717850 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f722000b7e0 con 0x7f72380b4000 2023-10-06T01:40:55.914 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.911+0000 7f7236ffd700 1 -- 172.21.15.124:0/2076717850 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722000bba0 con 0x7f72380b4000 2023-10-06T01:40:55.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.912+0000 7f723dadf700 1 -- 172.21.15.124:0/2076717850 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f72380b4000 msgr2=0x7f7238159b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:55.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.912+0000 7f723dadf700 1 --2- 172.21.15.124:0/2076717850 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f72380b4000 0x7f7238159b50 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7220009910 tx=0x7f72200091a0 comp rx=0 tx=0).stop 2023-10-06T01:40:55.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.912+0000 7f723dadf700 1 -- 172.21.15.124:0/2076717850 shutdown_connections 2023-10-06T01:40:55.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.912+0000 7f723dadf700 1 --2- 172.21.15.124:0/2076717850 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f72380b4000 0x7f7238159b50 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:55.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.912+0000 7f723dadf700 1 -- 172.21.15.124:0/2076717850 >> 172.21.15.124:0/2076717850 conn(0x7f723806d890 msgr2=0x7f723806dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:40:55.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.912+0000 7f723dadf700 1 -- 172.21.15.124:0/2076717850 shutdown_connections 2023-10-06T01:40:55.915 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.912+0000 7f723dadf700 1 -- 172.21.15.124:0/2076717850 wait complete. 2023-10-06T01:40:55.917 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.913+0000 7f723dadf700 1 Processor -- start 2023-10-06T01:40:55.917 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.913+0000 7f723dadf700 1 -- start start 2023-10-06T01:40:55.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.914+0000 7f723dadf700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f72380b1690 0x7f72380c8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:55.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.914+0000 7f723dadf700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f72380b4000 0x7f723816a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:55.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.914+0000 7f723dadf700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f723815cb10 0x7f72380cc390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:55.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.914+0000 7f723dadf700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72380ca880 con 0x7f723815cb10 2023-10-06T01:40:55.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.914+0000 7f723dadf700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723816ae60 con 0x7f72380b1690 2023-10-06T01:40:55.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.914+0000 7f723dadf700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72380c7b60 con 0x7f72380b4000 2023-10-06T01:40:55.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.914+0000 7f72377fe700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f72380b1690 0x7f72380c8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:55.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.914+0000 7f722ffff700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f72380b4000 0x7f723816a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:55.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.914+0000 7f722ffff700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f72380b4000 0x7f723816a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.124:43382/0 (socket says 172.21.15.124:43382) 2023-10-06T01:40:55.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.914+0000 7f722ffff700 1 -- 172.21.15.124:0/1387738393 learned_addr learned my addr 172.21.15.124:0/1387738393 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-10-06T01:40:55.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.914+0000 7f7237fff700 1 --2- 172.21.15.124:0/1387738393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f723815cb10 0x7f72380cc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:55.918 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.914+0000 7f72377fe700 1 -- 172.21.15.124:0/1387738393 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f72380b4000 msgr2=0x7f723816a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:55.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.914+0000 7f72377fe700 1 --2- 172.21.15.124:0/1387738393 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f72380b4000 0x7f723816a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:55.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.914+0000 7f72377fe700 1 -- 172.21.15.124:0/1387738393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f723815cb10 msgr2=0x7f72380cc390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:55.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.914+0000 7f72377fe700 1 --2- 172.21.15.124:0/1387738393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f723815cb10 0x7f72380cc390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:55.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.914+0000 7f72377fe700 1 -- 172.21.15.124:0/1387738393 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7220009420 con 0x7f72380b1690 2023-10-06T01:40:55.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.914+0000 7f722ffff700 1 --2- 172.21.15.124:0/1387738393 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f72380b4000 0x7f723816a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:40:55.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.914+0000 7f72377fe700 1 --2- 172.21.15.124:0/1387738393 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f72380b1690 0x7f72380c8c00 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f722800c560 tx=0x7f7228007d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:40:55.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.914+0000 7f722f7fe700 1 -- 172.21.15.124:0/1387738393 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7228005b30 con 0x7f72380b1690 2023-10-06T01:40:55.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.914+0000 7f722f7fe700 1 -- 172.21.15.124:0/1387738393 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7228005cb0 con 0x7f72380b1690 2023-10-06T01:40:55.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.914+0000 7f723dadf700 1 -- 172.21.15.124:0/1387738393 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72380c7e40 con 0x7f72380b1690 2023-10-06T01:40:55.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.914+0000 7f722f7fe700 1 -- 172.21.15.124:0/1387738393 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7228007320 con 0x7f72380b1690 2023-10-06T01:40:55.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.915+0000 7f723dadf700 1 -- 172.21.15.124:0/1387738393 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72380c8400 con 0x7f72380b1690 2023-10-06T01:40:55.919 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.915+0000 7f722f7fe700 1 -- 172.21.15.124:0/1387738393 <== mon.1 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f722801c6c0 con 0x7f72380b1690 2023-10-06T01:40:55.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.915+0000 7f722f7fe700 1 -- 172.21.15.124:0/1387738393 <== mon.1 v2:172.21.15.124:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7f7228005e20 con 0x7f72380b1690 2023-10-06T01:40:55.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.915+0000 7f723dadf700 1 -- 172.21.15.124:0/1387738393 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7238064050 con 0x7f72380b1690 2023-10-06T01:40:55.920 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:55.919+0000 7f722f7fe700 1 -- 172.21.15.124:0/1387738393 <== mon.1 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f7228017070 con 0x7f72380b1690 2023-10-06T01:40:55.927 INFO:tasks.ceph.mgr.x.smithi124.stderr:2023-10-06T01:40:55.926+0000 7fdf65db9180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-06T01:40:56.002 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.001+0000 7f723dadf700 1 -- 172.21.15.124:0/1387738393 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd new", "uuid": "438bb2d3-34e2-4b7c-9734-8360c1b67650", "id": 3} v 0) v1 -- 0x7f7238069570 con 0x7f72380b1690 2023-10-06T01:40:56.006 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.005+0000 7f722f7fe700 1 -- 172.21.15.124:0/1387738393 <== mon.1 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "438bb2d3-34e2-4b7c-9734-8360c1b67650", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f722801cba0 con 0x7f72380b1690 2023-10-06T01:40:56.006 INFO:teuthology.orchestra.run.smithi124.stdout:3 2023-10-06T01:40:56.007 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.006+0000 7f723dadf700 1 -- 172.21.15.124:0/1387738393 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f72380b1690 msgr2=0x7f72380c8c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:56.008 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.006+0000 7f723dadf700 1 --2- 172.21.15.124:0/1387738393 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f72380b1690 0x7f72380c8c00 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f722800c560 tx=0x7f7228007d50 comp rx=0 tx=0).stop 2023-10-06T01:40:56.008 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.006+0000 7f723dadf700 1 -- 172.21.15.124:0/1387738393 shutdown_connections 2023-10-06T01:40:56.008 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.006+0000 7f723dadf700 1 --2- 172.21.15.124:0/1387738393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f723815cb10 0x7f72380cc390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.008 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.006+0000 7f723dadf700 1 --2- 172.21.15.124:0/1387738393 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f72380b1690 0x7f72380c8c00 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.008 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.006+0000 7f723dadf700 1 --2- 172.21.15.124:0/1387738393 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f72380b4000 0x7f723816a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.008 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.006+0000 7f723dadf700 1 -- 172.21.15.124:0/1387738393 >> 172.21.15.124:0/1387738393 conn(0x7f723806d890 msgr2=0x7f723815bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:40:56.008 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.006+0000 7f723dadf700 1 -- 172.21.15.124:0/1387738393 shutdown_connections 2023-10-06T01:40:56.008 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.006+0000 7f723dadf700 1 -- 172.21.15.124:0/1387738393 wait complete. 2023-10-06T01:40:56.018 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --cluster ceph osd new e1956a8e-e0f8-4e7c-acbb-40037415a354 4 2023-10-06T01:40:56.097 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-10-06T01:40:56.096+0000 7fc8049c8180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-06T01:40:56.118 INFO:tasks.ceph.mgr.x.smithi124.stderr:2023-10-06T01:40:56.117+0000 7fdf65db9180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-06T01:40:56.133 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.132+0000 7f5db26fa700 1 Processor -- start 2023-10-06T01:40:56.133 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.132+0000 7f5db26fa700 1 -- start start 2023-10-06T01:40:56.136 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.133+0000 7f5db26fa700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5dac06b800 0x7f5dac15a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:56.136 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.133+0000 7f5db26fa700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f5dac06f420 0x7f5dac15a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:56.136 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.133+0000 7f5db26fa700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dac06aad0 con 0x7f5dac153d30 2023-10-06T01:40:56.136 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.133+0000 7f5db26fa700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dac0b0200 con 0x7f5dac06b800 2023-10-06T01:40:56.136 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.133+0000 7f5db26fa700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dac06b370 con 0x7f5dac06f420 2023-10-06T01:40:56.136 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.133+0000 7f5db1ef9700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f5dac06f420 0x7f5dac15a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:56.136 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.133+0000 7f5db1ef9700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f5dac06f420 0x7f5dac15a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:59282/0 (socket says 172.21.15.124:59282) 2023-10-06T01:40:56.136 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.133+0000 7f5db1ef9700 1 -- 172.21.15.124:0/1504858112 learned_addr learned my addr 172.21.15.124:0/1504858112 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-10-06T01:40:56.136 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.133+0000 7f5db0ef7700 1 --2- 172.21.15.124:0/1504858112 >> v2:172.21.15.27:3300/0 conn(0x7f5dac06b800 0x7f5dac15a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:56.136 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.133+0000 7f5db1ef9700 1 -- 172.21.15.124:0/1504858112 >> v2:172.21.15.27:3300/0 conn(0x7f5dac06b800 msgr2=0x7f5dac15a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:40:56.137 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.133+0000 7f5db1ef9700 1 --2- 172.21.15.124:0/1504858112 >> v2:172.21.15.27:3300/0 conn(0x7f5dac06b800 0x7f5dac15a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.137 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.133+0000 7f5db1ef9700 1 -- 172.21.15.124:0/1504858112 >> v1:172.21.15.27:6789/0 conn(0x7f5dac153d30 legacy=0x7f5dac06ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:56.137 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.133+0000 7f5db1ef9700 1 -- 172.21.15.124:0/1504858112 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dac15aa80 con 0x7f5dac06f420 2023-10-06T01:40:56.137 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.133+0000 7f5db1ef9700 1 --2- 172.21.15.124:0/1504858112 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5dac06f420 0x7f5dac15a540 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5dac0c9800 tx=0x7f5da4009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47e944c1279df326 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:40:56.137 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.133+0000 7f5da3fff700 1 -- 172.21.15.124:0/1504858112 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da400e070 con 0x7f5dac06f420 2023-10-06T01:40:56.137 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.133+0000 7f5da3fff700 1 -- 172.21.15.124:0/1504858112 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5da400bab0 con 0x7f5dac06f420 2023-10-06T01:40:56.137 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.133+0000 7f5da3fff700 1 -- 172.21.15.124:0/1504858112 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da40092e0 con 0x7f5dac06f420 2023-10-06T01:40:56.137 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.134+0000 7f5db26fa700 1 -- 172.21.15.124:0/1504858112 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5dac06f420 msgr2=0x7f5dac15a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:56.137 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.134+0000 7f5db26fa700 1 --2- 172.21.15.124:0/1504858112 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5dac06f420 0x7f5dac15a540 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5dac0c9800 tx=0x7f5da4009840 comp rx=0 tx=0).stop 2023-10-06T01:40:56.137 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.134+0000 7f5db26fa700 1 -- 172.21.15.124:0/1504858112 shutdown_connections 2023-10-06T01:40:56.137 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.134+0000 7f5db26fa700 1 --2- 172.21.15.124:0/1504858112 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5dac06f420 0x7f5dac15a540 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.138 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.134+0000 7f5db26fa700 1 --2- 172.21.15.124:0/1504858112 >> v2:172.21.15.27:3300/0 conn(0x7f5dac06b800 0x7f5dac15a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.138 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.134+0000 7f5db26fa700 1 -- 172.21.15.124:0/1504858112 >> 172.21.15.124:0/1504858112 conn(0x7f5dac0ca7b0 msgr2=0x7f5dac06e600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:40:56.138 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.134+0000 7f5db26fa700 1 -- 172.21.15.124:0/1504858112 shutdown_connections 2023-10-06T01:40:56.138 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.134+0000 7f5db26fa700 1 -- 172.21.15.124:0/1504858112 wait complete. 2023-10-06T01:40:56.140 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.135+0000 7f5db26fa700 1 Processor -- start 2023-10-06T01:40:56.140 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.135+0000 7f5db26fa700 1 -- start start 2023-10-06T01:40:56.140 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.136+0000 7f5db26fa700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5dac06b800 0x7f5dac0c5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:56.140 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.136+0000 7f5db26fa700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5dac06f420 0x7f5dac0c8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:56.140 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.136+0000 7f5db26fa700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5dac153d30 0x7f5dac06bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:56.140 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.136+0000 7f5db26fa700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dac0c8390 con 0x7f5dac06b800 2023-10-06T01:40:56.140 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.136+0000 7f5db26fa700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dac0c7ac0 con 0x7f5dac153d30 2023-10-06T01:40:56.140 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.136+0000 7f5db26fa700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dac16a680 con 0x7f5dac06f420 2023-10-06T01:40:56.141 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.136+0000 7f5db0ef7700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5dac06f420 0x7f5dac0c8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:56.141 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.136+0000 7f5db1ef9700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5dac153d30 0x7f5dac06bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:56.141 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.136+0000 7f5db0ef7700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5dac06f420 0x7f5dac0c8820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.124:43394/0 (socket says 172.21.15.124:43394) 2023-10-06T01:40:56.141 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.136+0000 7f5db1ef9700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5dac153d30 0x7f5dac06bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:59288/0 (socket says 172.21.15.124:59288) 2023-10-06T01:40:56.141 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.136+0000 7f5db0ef7700 1 -- 172.21.15.124:0/1758371210 learned_addr learned my addr 172.21.15.124:0/1758371210 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-10-06T01:40:56.141 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.136+0000 7f5db16f8700 1 --2- 172.21.15.124:0/1758371210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5dac06b800 0x7f5dac0c5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:56.141 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.137+0000 7f5db0ef7700 1 -- 172.21.15.124:0/1758371210 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5dac153d30 msgr2=0x7f5dac06bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:56.141 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.137+0000 7f5db0ef7700 1 --2- 172.21.15.124:0/1758371210 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5dac153d30 0x7f5dac06bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.141 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.137+0000 7f5db0ef7700 1 -- 172.21.15.124:0/1758371210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5dac06b800 msgr2=0x7f5dac0c5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:56.141 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.137+0000 7f5db0ef7700 1 --2- 172.21.15.124:0/1758371210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5dac06b800 0x7f5dac0c5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.141 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.137+0000 7f5db0ef7700 1 -- 172.21.15.124:0/1758371210 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da4009ae0 con 0x7f5dac06f420 2023-10-06T01:40:56.141 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.137+0000 7f5db0ef7700 1 --2- 172.21.15.124:0/1758371210 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5dac06f420 0x7f5dac0c8820 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c00e930 tx=0x7f5d9c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:40:56.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.137+0000 7f5da17fa700 1 -- 172.21.15.124:0/1758371210 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d9c013680 con 0x7f5dac06f420 2023-10-06T01:40:56.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.137+0000 7f5da17fa700 1 -- 172.21.15.124:0/1758371210 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d9c0137e0 con 0x7f5dac06f420 2023-10-06T01:40:56.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.137+0000 7f5da17fa700 1 -- 172.21.15.124:0/1758371210 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d9c013ba0 con 0x7f5dac06f420 2023-10-06T01:40:56.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.137+0000 7f5db26fa700 1 -- 172.21.15.124:0/1758371210 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dac0c7d40 con 0x7f5dac06f420 2023-10-06T01:40:56.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.137+0000 7f5db26fa700 1 -- 172.21.15.124:0/1758371210 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dac0c5770 con 0x7f5dac06f420 2023-10-06T01:40:56.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.137+0000 7f5da17fa700 1 -- 172.21.15.124:0/1758371210 <== mon.2 v2:172.21.15.124:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f5d9c005c80 con 0x7f5dac06f420 2023-10-06T01:40:56.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.137+0000 7f5da17fa700 1 -- 172.21.15.124:0/1758371210 <== mon.2 v2:172.21.15.124:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7f5d9c0065c0 con 0x7f5dac06f420 2023-10-06T01:40:56.142 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.138+0000 7f5db26fa700 1 -- 172.21.15.124:0/1758371210 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dac0c8540 con 0x7f5dac06f420 2023-10-06T01:40:56.143 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.140+0000 7f5da17fa700 1 -- 172.21.15.124:0/1758371210 <== mon.2 v2:172.21.15.124:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f5dac0c8540 con 0x7f5dac06f420 2023-10-06T01:40:56.224 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.223+0000 7f5db26fa700 1 -- 172.21.15.124:0/1758371210 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_command({"prefix": "osd new", "uuid": "e1956a8e-e0f8-4e7c-acbb-40037415a354", "id": 4} v 0) v1 -- 0x7f5dac04c3c0 con 0x7f5dac06f420 2023-10-06T01:40:56.227 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.226+0000 7f5da17fa700 1 -- 172.21.15.124:0/1758371210 <== mon.2 v2:172.21.15.124:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e1956a8e-e0f8-4e7c-acbb-40037415a354", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5d9c021060 con 0x7f5dac06f420 2023-10-06T01:40:56.228 INFO:teuthology.orchestra.run.smithi124.stdout:4 2023-10-06T01:40:56.229 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.227+0000 7f5db26fa700 1 -- 172.21.15.124:0/1758371210 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5dac06f420 msgr2=0x7f5dac0c8820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:56.229 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.227+0000 7f5db26fa700 1 --2- 172.21.15.124:0/1758371210 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5dac06f420 0x7f5dac0c8820 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f5d9c00e930 tx=0x7f5d9c0091a0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.229 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.227+0000 7f5db26fa700 1 -- 172.21.15.124:0/1758371210 shutdown_connections 2023-10-06T01:40:56.229 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.227+0000 7f5db26fa700 1 --2- 172.21.15.124:0/1758371210 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5dac153d30 0x7f5dac06bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.229 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.227+0000 7f5db26fa700 1 --2- 172.21.15.124:0/1758371210 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5dac06b800 0x7f5dac0c5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.229 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.227+0000 7f5db26fa700 1 --2- 172.21.15.124:0/1758371210 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5dac06f420 0x7f5dac0c8820 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.229 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.227+0000 7f5db26fa700 1 -- 172.21.15.124:0/1758371210 >> 172.21.15.124:0/1758371210 conn(0x7f5dac0ca7b0 msgr2=0x7f5dac152d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:40:56.229 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.227+0000 7f5db26fa700 1 -- 172.21.15.124:0/1758371210 shutdown_connections 2023-10-06T01:40:56.229 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.228+0000 7f5db26fa700 1 -- 172.21.15.124:0/1758371210 wait complete. 2023-10-06T01:40:56.240 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --cluster ceph osd new 306c59d4-846d-4b73-b1c9-051bec9e9368 5 2023-10-06T01:40:56.277 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-10-06T01:40:56.276+0000 7fc8049c8180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-06T01:40:56.299 INFO:tasks.ceph.mgr.x.smithi124.stderr:2023-10-06T01:40:56.298+0000 7fdf65db9180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-06T01:40:56.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.353+0000 7f3e58d89700 1 Processor -- start 2023-10-06T01:40:56.354 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.353+0000 7f3e58d89700 1 -- start start 2023-10-06T01:40:56.357 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.354+0000 7f3e58d89700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3e54159500 0x7f3e541598d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:56.357 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.354+0000 7f3e58d89700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3e5415dab0 0x7f3e540cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:56.357 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.354+0000 7f3e58d89700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e540b1110 con 0x7f3e540b3fd0 2023-10-06T01:40:56.357 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.354+0000 7f3e58d89700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e540b13b0 con 0x7f3e54159500 2023-10-06T01:40:56.357 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.354+0000 7f3e58d89700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e540b1690 con 0x7f3e5415dab0 2023-10-06T01:40:56.357 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.354+0000 7f3e52d9d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3e5415dab0 0x7f3e540cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:56.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.354+0000 7f3e52d9d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3e5415dab0 0x7f3e540cf6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.124:43408/0 (socket says 172.21.15.124:43408) 2023-10-06T01:40:56.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.354+0000 7f3e52d9d700 1 -- 172.21.15.124:0/3722644989 learned_addr learned my addr 172.21.15.124:0/3722644989 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-10-06T01:40:56.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.354+0000 7f3e5259c700 1 --2- 172.21.15.124:0/3722644989 >> v2:172.21.15.27:3300/0 conn(0x7f3e54159500 0x7f3e541598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:56.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.355+0000 7f3e52d9d700 1 -- 172.21.15.124:0/3722644989 >> v2:172.21.15.27:3300/0 conn(0x7f3e54159500 msgr2=0x7f3e541598d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:40:56.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.355+0000 7f3e52d9d700 1 --2- 172.21.15.124:0/3722644989 >> v2:172.21.15.27:3300/0 conn(0x7f3e54159500 0x7f3e541598d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.355+0000 7f3e52d9d700 1 -- 172.21.15.124:0/3722644989 >> v1:172.21.15.27:6789/0 conn(0x7f3e540b3fd0 legacy=0x7f3e5415d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:56.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.355+0000 7f3e52d9d700 1 -- 172.21.15.124:0/3722644989 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e540b1970 con 0x7f3e5415dab0 2023-10-06T01:40:56.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.355+0000 7f3e52d9d700 1 --2- 172.21.15.124:0/3722644989 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3e5415dab0 0x7f3e540cf6c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c009f20 tx=0x7f3e4c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63bc6125ba2bf3e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:40:56.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.355+0000 7f3e5159a700 1 -- 172.21.15.124:0/3722644989 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c00e070 con 0x7f3e5415dab0 2023-10-06T01:40:56.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.355+0000 7f3e5159a700 1 -- 172.21.15.124:0/3722644989 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e4c00b9d0 con 0x7f3e5415dab0 2023-10-06T01:40:56.358 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.355+0000 7f3e5159a700 1 -- 172.21.15.124:0/3722644989 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c0092e0 con 0x7f3e5415dab0 2023-10-06T01:40:56.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.356+0000 7f3e58d89700 1 -- 172.21.15.124:0/3722644989 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3e5415dab0 msgr2=0x7f3e540cf6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:56.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.356+0000 7f3e58d89700 1 --2- 172.21.15.124:0/3722644989 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3e5415dab0 0x7f3e540cf6c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c009f20 tx=0x7f3e4c009840 comp rx=0 tx=0).stop 2023-10-06T01:40:56.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.356+0000 7f3e58d89700 1 -- 172.21.15.124:0/3722644989 shutdown_connections 2023-10-06T01:40:56.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.356+0000 7f3e58d89700 1 --2- 172.21.15.124:0/3722644989 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3e5415dab0 0x7f3e540cf6c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.356+0000 7f3e58d89700 1 --2- 172.21.15.124:0/3722644989 >> v2:172.21.15.27:3300/0 conn(0x7f3e54159500 0x7f3e541598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.356+0000 7f3e58d89700 1 -- 172.21.15.124:0/3722644989 >> 172.21.15.124:0/3722644989 conn(0x7f3e54067630 msgr2=0x7f3e54069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:40:56.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.356+0000 7f3e58d89700 1 -- 172.21.15.124:0/3722644989 shutdown_connections 2023-10-06T01:40:56.359 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.356+0000 7f3e58d89700 1 -- 172.21.15.124:0/3722644989 wait complete. 2023-10-06T01:40:56.360 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-10-06T01:40:56.356+0000 7fc8049c8180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-06T01:40:56.361 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.357+0000 7f3e58d89700 1 Processor -- start 2023-10-06T01:40:56.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.357+0000 7f3e58d89700 1 -- start start 2023-10-06T01:40:56.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.357+0000 7f3e58d89700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e540b3fd0 0x7f3e541699a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:56.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.357+0000 7f3e58d89700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3e54159500 0x7f3e54169ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:56.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.358+0000 7f3e58d89700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3e5415dab0 0x7f3e54149500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:56.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.358+0000 7f3e58d89700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5416e8b0 con 0x7f3e540b3fd0 2023-10-06T01:40:56.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.358+0000 7f3e58d89700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5416ae50 con 0x7f3e54159500 2023-10-06T01:40:56.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.358+0000 7f3e58d89700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5416b2e0 con 0x7f3e5415dab0 2023-10-06T01:40:56.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.358+0000 7f3e52d9d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3e5415dab0 0x7f3e54149500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:56.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.358+0000 7f3e51d9b700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3e54159500 0x7f3e54169ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:56.362 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.358+0000 7f3e52d9d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3e5415dab0 0x7f3e54149500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.124:43412/0 (socket says 172.21.15.124:43412) 2023-10-06T01:40:56.363 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.358+0000 7f3e52d9d700 1 -- 172.21.15.124:0/662820046 learned_addr learned my addr 172.21.15.124:0/662820046 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-10-06T01:40:56.363 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.358+0000 7f3e5259c700 1 --2- 172.21.15.124:0/662820046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e540b3fd0 0x7f3e541699a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:56.363 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.358+0000 7f3e52d9d700 1 -- 172.21.15.124:0/662820046 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3e54159500 msgr2=0x7f3e54169ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:56.363 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.358+0000 7f3e52d9d700 1 --2- 172.21.15.124:0/662820046 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3e54159500 0x7f3e54169ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.363 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.358+0000 7f3e52d9d700 1 -- 172.21.15.124:0/662820046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e540b3fd0 msgr2=0x7f3e541699a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:56.363 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.358+0000 7f3e52d9d700 1 --2- 172.21.15.124:0/662820046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e540b3fd0 0x7f3e541699a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.363 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.358+0000 7f3e52d9d700 1 -- 172.21.15.124:0/662820046 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e4c009ae0 con 0x7f3e5415dab0 2023-10-06T01:40:56.363 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.358+0000 7f3e52d9d700 1 --2- 172.21.15.124:0/662820046 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3e5415dab0 0x7f3e54149500 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c00b610 tx=0x7f3e4c018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:40:56.363 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.358+0000 7f3e427fc700 1 -- 172.21.15.124:0/662820046 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c008ad0 con 0x7f3e5415dab0 2023-10-06T01:40:56.363 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.358+0000 7f3e427fc700 1 -- 172.21.15.124:0/662820046 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e4c008c30 con 0x7f3e5415dab0 2023-10-06T01:40:56.363 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.358+0000 7f3e58d89700 1 -- 172.21.15.124:0/662820046 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e5416a520 con 0x7f3e5415dab0 2023-10-06T01:40:56.363 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.358+0000 7f3e427fc700 1 -- 172.21.15.124:0/662820046 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c026460 con 0x7f3e5415dab0 2023-10-06T01:40:56.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.358+0000 7f3e58d89700 1 -- 172.21.15.124:0/662820046 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e5416aae0 con 0x7f3e5415dab0 2023-10-06T01:40:56.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.358+0000 7f3e427fc700 1 -- 172.21.15.124:0/662820046 <== mon.2 v2:172.21.15.124:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f3e4c003040 con 0x7f3e5415dab0 2023-10-06T01:40:56.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.358+0000 7f3e427fc700 1 -- 172.21.15.124:0/662820046 <== mon.2 v2:172.21.15.124:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1296+0+0 (secure 0 0 0) 0x7f3e4c026cd0 con 0x7f3e5415dab0 2023-10-06T01:40:56.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.359+0000 7f3e58d89700 1 -- 172.21.15.124:0/662820046 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e5416b000 con 0x7f3e5415dab0 2023-10-06T01:40:56.364 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.361+0000 7f3e427fc700 1 -- 172.21.15.124:0/662820046 <== mon.2 v2:172.21.15.124:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f3e4c0466d0 con 0x7f3e5415dab0 2023-10-06T01:40:56.379 INFO:tasks.ceph.mgr.x.smithi124.stderr:2023-10-06T01:40:56.378+0000 7fdf65db9180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-06T01:40:56.445 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.444+0000 7f3e58d89700 1 -- 172.21.15.124:0/662820046 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_command({"prefix": "osd new", "uuid": "306c59d4-846d-4b73-b1c9-051bec9e9368", "id": 5} v 0) v1 -- 0x7f3e54064050 con 0x7f3e5415dab0 2023-10-06T01:40:56.448 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.447+0000 7f3e427fc700 1 -- 172.21.15.124:0/662820046 <== mon.2 v2:172.21.15.124:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "306c59d4-846d-4b73-b1c9-051bec9e9368", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3e4c035030 con 0x7f3e5415dab0 2023-10-06T01:40:56.449 INFO:teuthology.orchestra.run.smithi124.stdout:5 2023-10-06T01:40:56.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.448+0000 7f3e58d89700 1 -- 172.21.15.124:0/662820046 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3e5415dab0 msgr2=0x7f3e54149500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:56.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.448+0000 7f3e58d89700 1 --2- 172.21.15.124:0/662820046 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3e5415dab0 0x7f3e54149500 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c00b610 tx=0x7f3e4c018000 comp rx=0 tx=0).stop 2023-10-06T01:40:56.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.448+0000 7f3e58d89700 1 -- 172.21.15.124:0/662820046 shutdown_connections 2023-10-06T01:40:56.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.448+0000 7f3e58d89700 1 --2- 172.21.15.124:0/662820046 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3e5415dab0 0x7f3e54149500 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.448+0000 7f3e58d89700 1 --2- 172.21.15.124:0/662820046 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3e540b3fd0 0x7f3e541699a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.448+0000 7f3e58d89700 1 --2- 172.21.15.124:0/662820046 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3e54159500 0x7f3e54169ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.448+0000 7f3e58d89700 1 -- 172.21.15.124:0/662820046 >> 172.21.15.124:0/662820046 conn(0x7f3e54067630 msgr2=0x7f3e540cfa20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:40:56.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.448+0000 7f3e58d89700 1 -- 172.21.15.124:0/662820046 shutdown_connections 2023-10-06T01:40:56.450 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.448+0000 7f3e58d89700 1 -- 172.21.15.124:0/662820046 wait complete. 2023-10-06T01:40:56.461 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --cluster ceph osd new 15463e53-ac8d-47d1-b4f6-c0d31f29e4a9 6 2023-10-06T01:40:56.496 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-10-06T01:40:56.495+0000 7fc8049c8180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-06T01:40:56.521 INFO:tasks.ceph.mgr.x.smithi124.stderr:2023-10-06T01:40:56.519+0000 7fdf65db9180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-06T01:40:56.578 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.577+0000 7fc6ba0fb700 1 Processor -- start 2023-10-06T01:40:56.578 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.577+0000 7fc6ba0fb700 1 -- start start 2023-10-06T01:40:56.582 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.578+0000 7fc6ba0fb700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc6b415dab0 0x7fc6b40cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:56.582 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.578+0000 7fc6ba0fb700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6b40b1110 con 0x7fc6b4159500 2023-10-06T01:40:56.582 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.578+0000 7fc6ba0fb700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6b40b13b0 con 0x7fc6b41598d0 2023-10-06T01:40:56.582 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.578+0000 7fc6ba0fb700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6b40b1690 con 0x7fc6b415dab0 2023-10-06T01:40:56.582 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.578+0000 7fc6b2ffd700 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fc6b41598d0 0x7fc6b415d3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.124:44092/0 (socket says 172.21.15.124:44092) 2023-10-06T01:40:56.582 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.578+0000 7fc6b2ffd700 1 -- 172.21.15.124:0/2968864907 learned_addr learned my addr 172.21.15.124:0/2968864907 (peer_addr_for_me v1:172.21.15.124:0/0) 2023-10-06T01:40:56.582 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.578+0000 7fc6b3fff700 1 --2- 172.21.15.124:0/2968864907 >> v2:172.21.15.27:3300/0 conn(0x7fc6b415dab0 0x7fc6b40cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:56.582 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.578+0000 7fc6b27fc700 1 -- 172.21.15.124:0/2968864907 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 741402719 0 0) 0x7fc6b40b13b0 con 0x7fc6b41598d0 2023-10-06T01:40:56.582 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.578+0000 7fc6b27fc700 1 -- 172.21.15.124:0/2968864907 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc69c003570 con 0x7fc6b41598d0 2023-10-06T01:40:56.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.578+0000 7fc6b27fc700 1 -- 172.21.15.124:0/2968864907 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2011802667 0 0) 0x7fc6b40b1110 con 0x7fc6b4159500 2023-10-06T01:40:56.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.578+0000 7fc6b27fc700 1 -- 172.21.15.124:0/2968864907 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6b40b13b0 con 0x7fc6b4159500 2023-10-06T01:40:56.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.578+0000 7fc6b27fc700 1 -- 172.21.15.124:0/2968864907 <== mon.1 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4011119405 0 0) 0x7fc69c003570 con 0x7fc6b41598d0 2023-10-06T01:40:56.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.578+0000 7fc6b27fc700 1 -- 172.21.15.124:0/2968864907 >> v2:172.21.15.27:3300/0 conn(0x7fc6b415dab0 msgr2=0x7fc6b40cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:40:56.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.578+0000 7fc6b27fc700 1 --2- 172.21.15.124:0/2968864907 >> v2:172.21.15.27:3300/0 conn(0x7fc6b415dab0 0x7fc6b40cf6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.578+0000 7fc6b27fc700 1 -- 172.21.15.124:0/2968864907 >> v1:172.21.15.27:6789/0 conn(0x7fc6b4159500 legacy=0x7fc6b40b3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:56.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.578+0000 7fc6b27fc700 1 -- 172.21.15.124:0/2968864907 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b40b1970 con 0x7fc6b41598d0 2023-10-06T01:40:56.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.578+0000 7fc6b27fc700 1 -- 172.21.15.124:0/2968864907 <== mon.1 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2619078944 0 0) 0x7fc6a8001470 con 0x7fc6b41598d0 2023-10-06T01:40:56.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.578+0000 7fc6b27fc700 1 -- 172.21.15.124:0/2968864907 >> v1:172.21.15.124:6789/0 conn(0x7fc6b41598d0 legacy=0x7fc6b415d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:56.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.578+0000 7fc6b27fc700 1 --2- 172.21.15.124:0/2968864907 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc69c003bf0 0x7fc69c003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:56.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.578+0000 7fc6b27fc700 1 --2- 172.21.15.124:0/2968864907 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc69c004810 0x7fc69c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:56.583 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.578+0000 7fc6b27fc700 1 --2- 172.21.15.124:0/2968864907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc69c0051c0 0x7fc69c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:56.584 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.578+0000 7fc6b27fc700 1 -- 172.21.15.124:0/2968864907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b40b1690 con 0x7fc69c0051c0 2023-10-06T01:40:56.584 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.578+0000 7fc6b27fc700 1 -- 172.21.15.124:0/2968864907 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc69c00c0b0 con 0x7fc69c004810 2023-10-06T01:40:56.584 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.578+0000 7fc6b27fc700 1 -- 172.21.15.124:0/2968864907 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc69c00c270 con 0x7fc69c003bf0 2023-10-06T01:40:56.584 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.578+0000 7fc6b2ffd700 1 --2- 172.21.15.124:0/2968864907 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc69c004810 0x7fc69c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:56.584 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.578+0000 7fc6b3fff700 1 --2- 172.21.15.124:0/2968864907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc69c0051c0 0x7fc69c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:56.584 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.578+0000 7fc6b37fe700 1 --2- 172.21.15.124:0/2968864907 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc69c003bf0 0x7fc69c003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:56.584 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.579+0000 7fc6b2ffd700 1 -- 172.21.15.124:0/2968864907 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc69c003bf0 msgr2=0x7fc69c003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:40:56.584 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.579+0000 7fc6b2ffd700 1 --2- 172.21.15.124:0/2968864907 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc69c003bf0 0x7fc69c003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.584 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.579+0000 7fc6b2ffd700 1 -- 172.21.15.124:0/2968864907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc69c0051c0 msgr2=0x7fc69c00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:40:56.584 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.579+0000 7fc6b2ffd700 1 --2- 172.21.15.124:0/2968864907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc69c0051c0 0x7fc69c00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.584 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.579+0000 7fc6b2ffd700 1 -- 172.21.15.124:0/2968864907 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc69c003570 con 0x7fc69c004810 2023-10-06T01:40:56.585 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.579+0000 7fc6b3fff700 1 -- 172.21.15.124:0/2968864907 reap_dead start 2023-10-06T01:40:56.585 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.579+0000 7fc6b2ffd700 1 --2- 172.21.15.124:0/2968864907 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc69c004810 0x7fc69c004c80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc6a800c2d0 tx=0x7fc6a8001e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be62879373de7b9d server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:40:56.585 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.579+0000 7fc6b27fc700 1 -- 172.21.15.124:0/2968864907 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a800cb60 con 0x7fc69c004810 2023-10-06T01:40:56.585 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.579+0000 7fc6b27fc700 1 -- 172.21.15.124:0/2968864907 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a800ccc0 con 0x7fc69c004810 2023-10-06T01:40:56.585 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.579+0000 7fc6b27fc700 1 -- 172.21.15.124:0/2968864907 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a8005b50 con 0x7fc69c004810 2023-10-06T01:40:56.585 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.579+0000 7fc6ba0fb700 1 -- 172.21.15.124:0/2968864907 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc69c004810 msgr2=0x7fc69c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:56.585 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.579+0000 7fc6ba0fb700 1 --2- 172.21.15.124:0/2968864907 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc69c004810 0x7fc69c004c80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fc6a800c2d0 tx=0x7fc6a8001e10 comp rx=0 tx=0).stop 2023-10-06T01:40:56.585 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.580+0000 7fc6ba0fb700 1 -- 172.21.15.124:0/2968864907 shutdown_connections 2023-10-06T01:40:56.585 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.580+0000 7fc6ba0fb700 1 --2- 172.21.15.124:0/2968864907 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc69c004810 0x7fc69c004c80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.585 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.580+0000 7fc6ba0fb700 1 -- 172.21.15.124:0/2968864907 >> 172.21.15.124:0/2968864907 conn(0x7fc6b4067630 msgr2=0x7fc6b4069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:40:56.585 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.580+0000 7fc6ba0fb700 1 -- 172.21.15.124:0/2968864907 shutdown_connections 2023-10-06T01:40:56.586 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.580+0000 7fc6ba0fb700 1 -- 172.21.15.124:0/2968864907 wait complete. 2023-10-06T01:40:56.586 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.581+0000 7fc6ba0fb700 1 Processor -- start 2023-10-06T01:40:56.586 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.581+0000 7fc6ba0fb700 1 -- start start 2023-10-06T01:40:56.588 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.581+0000 7fc6ba0fb700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc69c004810 0x7fc6b416afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:56.588 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.581+0000 7fc6ba0fb700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6b4150780 0x7fc6b413df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:56.588 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.581+0000 7fc6ba0fb700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc6b414de20 0x7fc6b414e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:56.588 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.582+0000 7fc6ba0fb700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b415a7a0 con 0x7fc6b4150780 2023-10-06T01:40:56.588 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.582+0000 7fc6ba0fb700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b415b600 con 0x7fc69c004810 2023-10-06T01:40:56.588 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.582+0000 7fc6ba0fb700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6b414e7b0 con 0x7fc6b414de20 2023-10-06T01:40:56.588 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.582+0000 7fc6b37fe700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc69c004810 0x7fc6b416afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:56.589 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.582+0000 7fc6b37fe700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc69c004810 0x7fc6b416afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:59322/0 (socket says 172.21.15.124:59322) 2023-10-06T01:40:56.589 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.582+0000 7fc6b37fe700 1 -- 172.21.15.124:0/61821440 learned_addr learned my addr 172.21.15.124:0/61821440 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-10-06T01:40:56.589 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.582+0000 7fc6b3fff700 1 --2- 172.21.15.124:0/61821440 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc6b414de20 0x7fc6b414e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:56.589 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.582+0000 7fc6b2ffd700 1 --2- 172.21.15.124:0/61821440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6b4150780 0x7fc6b413df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:56.589 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.582+0000 7fc6b37fe700 1 -- 172.21.15.124:0/61821440 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc6b414de20 msgr2=0x7fc6b414e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:56.589 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.582+0000 7fc6b37fe700 1 --2- 172.21.15.124:0/61821440 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc6b414de20 0x7fc6b414e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.589 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.582+0000 7fc6b37fe700 1 -- 172.21.15.124:0/61821440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6b4150780 msgr2=0x7fc6b413df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:56.589 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.582+0000 7fc6b37fe700 1 --2- 172.21.15.124:0/61821440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6b4150780 0x7fc6b413df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.589 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.582+0000 7fc6b37fe700 1 -- 172.21.15.124:0/61821440 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b415a9f0 con 0x7fc69c004810 2023-10-06T01:40:56.589 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.582+0000 7fc6b37fe700 1 --2- 172.21.15.124:0/61821440 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc69c004810 0x7fc6b416afc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4002c30 tx=0x7fc6a4007d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:40:56.589 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.582+0000 7fc69b7fe700 1 -- 172.21.15.124:0/61821440 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a400f920 con 0x7fc69c004810 2023-10-06T01:40:56.589 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.582+0000 7fc69b7fe700 1 -- 172.21.15.124:0/61821440 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a400fa80 con 0x7fc69c004810 2023-10-06T01:40:56.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.582+0000 7fc6ba0fb700 1 -- 172.21.15.124:0/61821440 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6b415acd0 con 0x7fc69c004810 2023-10-06T01:40:56.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.582+0000 7fc69b7fe700 1 -- 172.21.15.124:0/61821440 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a400fe40 con 0x7fc69c004810 2023-10-06T01:40:56.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.582+0000 7fc6ba0fb700 1 -- 172.21.15.124:0/61821440 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6b415b290 con 0x7fc69c004810 2023-10-06T01:40:56.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.582+0000 7fc69b7fe700 1 -- 172.21.15.124:0/61821440 <== mon.1 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fc6a4013880 con 0x7fc69c004810 2023-10-06T01:40:56.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.582+0000 7fc69b7fe700 1 -- 172.21.15.124:0/61821440 <== mon.1 v2:172.21.15.124:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7fc6a401cb40 con 0x7fc69c004810 2023-10-06T01:40:56.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.583+0000 7fc6ba0fb700 1 -- 172.21.15.124:0/61821440 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6b4064050 con 0x7fc69c004810 2023-10-06T01:40:56.590 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.585+0000 7fc69b7fe700 1 -- 172.21.15.124:0/61821440 <== mon.1 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fc6a403b6d0 con 0x7fc69c004810 2023-10-06T01:40:56.595 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-10-06T01:40:56.594+0000 7fc8049c8180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-06T01:40:56.620 INFO:tasks.ceph.mgr.x.smithi124.stderr:2023-10-06T01:40:56.619+0000 7fdf65db9180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-06T01:40:56.668 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.666+0000 7fc6ba0fb700 1 -- 172.21.15.124:0/61821440 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd new", "uuid": "15463e53-ac8d-47d1-b4f6-c0d31f29e4a9", "id": 6} v 0) v1 -- 0x7fc6b40d0420 con 0x7fc69c004810 2023-10-06T01:40:56.671 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.670+0000 7fc69b7fe700 1 -- 172.21.15.124:0/61821440 <== mon.1 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "15463e53-ac8d-47d1-b4f6-c0d31f29e4a9", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc6a402a070 con 0x7fc69c004810 2023-10-06T01:40:56.671 INFO:teuthology.orchestra.run.smithi124.stdout:6 2023-10-06T01:40:56.672 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.671+0000 7fc6ba0fb700 1 -- 172.21.15.124:0/61821440 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc69c004810 msgr2=0x7fc6b416afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:56.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.671+0000 7fc6ba0fb700 1 --2- 172.21.15.124:0/61821440 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc69c004810 0x7fc6b416afc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4002c30 tx=0x7fc6a4007d30 comp rx=0 tx=0).stop 2023-10-06T01:40:56.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.671+0000 7fc6ba0fb700 1 -- 172.21.15.124:0/61821440 shutdown_connections 2023-10-06T01:40:56.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.671+0000 7fc6ba0fb700 1 --2- 172.21.15.124:0/61821440 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc6b414de20 0x7fc6b414e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.671+0000 7fc6ba0fb700 1 --2- 172.21.15.124:0/61821440 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc69c004810 0x7fc6b416afc0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.671+0000 7fc6ba0fb700 1 --2- 172.21.15.124:0/61821440 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6b4150780 0x7fc6b413df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.671+0000 7fc6ba0fb700 1 -- 172.21.15.124:0/61821440 >> 172.21.15.124:0/61821440 conn(0x7fc6b4067630 msgr2=0x7fc6b4156360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:40:56.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.671+0000 7fc6ba0fb700 1 -- 172.21.15.124:0/61821440 shutdown_connections 2023-10-06T01:40:56.673 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.671+0000 7fc6ba0fb700 1 -- 172.21.15.124:0/61821440 wait complete. 2023-10-06T01:40:56.684 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --cluster ceph osd new ae463a90-a9d3-498f-86fa-47a35b3fe889 7 2023-10-06T01:40:56.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.797+0000 7f67a3b73700 1 Processor -- start 2023-10-06T01:40:56.799 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.797+0000 7f67a3b73700 1 -- start start 2023-10-06T01:40:56.801 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.798+0000 7f67a3b73700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f679c159500 0x7f679c1598d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:56.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.798+0000 7f67a3b73700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f679c15dab0 0x7f679c0cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:56.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.798+0000 7f67a3b73700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f679c0b1110 con 0x7f679c0b3fd0 2023-10-06T01:40:56.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.798+0000 7f67a3b73700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f679c0b13b0 con 0x7f679c159500 2023-10-06T01:40:56.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.798+0000 7f67a3b73700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f679c0b1690 con 0x7f679c15dab0 2023-10-06T01:40:56.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.798+0000 7f67a2110700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f679c15dab0 0x7f679c0cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:56.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.798+0000 7f67a2110700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f679c15dab0 0x7f679c0cf6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.124:59332/0 (socket says 172.21.15.124:59332) 2023-10-06T01:40:56.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.798+0000 7f67a2110700 1 -- 172.21.15.124:0/1275599559 learned_addr learned my addr 172.21.15.124:0/1275599559 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-10-06T01:40:56.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.798+0000 7f67a190f700 1 --2- 172.21.15.124:0/1275599559 >> v2:172.21.15.27:3300/0 conn(0x7f679c159500 0x7f679c1598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:56.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.799+0000 7f67a2110700 1 -- 172.21.15.124:0/1275599559 >> v2:172.21.15.27:3300/0 conn(0x7f679c159500 msgr2=0x7f679c1598d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:40:56.802 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.799+0000 7f67a2110700 1 --2- 172.21.15.124:0/1275599559 >> v2:172.21.15.27:3300/0 conn(0x7f679c159500 0x7f679c1598d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.803 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.799+0000 7f67a2110700 1 -- 172.21.15.124:0/1275599559 >> v1:172.21.15.27:6789/0 conn(0x7f679c0b3fd0 legacy=0x7f679c15d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:56.803 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.799+0000 7f67a2110700 1 -- 172.21.15.124:0/1275599559 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f679c0b1970 con 0x7f679c15dab0 2023-10-06T01:40:56.803 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.799+0000 7f67a2110700 1 --2- 172.21.15.124:0/1275599559 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f679c15dab0 0x7f679c0cf6c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f679c06a270 tx=0x7f6794009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f72a6d05a8116567 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:40:56.803 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.799+0000 7f67a090d700 1 -- 172.21.15.124:0/1275599559 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679400e070 con 0x7f679c15dab0 2023-10-06T01:40:56.803 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.799+0000 7f67a090d700 1 -- 172.21.15.124:0/1275599559 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f679400ba20 con 0x7f679c15dab0 2023-10-06T01:40:56.803 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.799+0000 7f67a090d700 1 -- 172.21.15.124:0/1275599559 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67940092e0 con 0x7f679c15dab0 2023-10-06T01:40:56.803 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.799+0000 7f67a3b73700 1 -- 172.21.15.124:0/1275599559 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f679c15dab0 msgr2=0x7f679c0cf6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:56.803 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.799+0000 7f67a3b73700 1 --2- 172.21.15.124:0/1275599559 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f679c15dab0 0x7f679c0cf6c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f679c06a270 tx=0x7f6794009840 comp rx=0 tx=0).stop 2023-10-06T01:40:56.803 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.799+0000 7f67a3b73700 1 -- 172.21.15.124:0/1275599559 shutdown_connections 2023-10-06T01:40:56.803 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.799+0000 7f67a3b73700 1 --2- 172.21.15.124:0/1275599559 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f679c15dab0 0x7f679c0cf6c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.804 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.799+0000 7f67a3b73700 1 --2- 172.21.15.124:0/1275599559 >> v2:172.21.15.27:3300/0 conn(0x7f679c159500 0x7f679c1598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.804 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.799+0000 7f67a3b73700 1 -- 172.21.15.124:0/1275599559 >> 172.21.15.124:0/1275599559 conn(0x7f679c067630 msgr2=0x7f679c069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:40:56.804 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.799+0000 7f67a3b73700 1 -- 172.21.15.124:0/1275599559 shutdown_connections 2023-10-06T01:40:56.804 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.800+0000 7f67a3b73700 1 -- 172.21.15.124:0/1275599559 wait complete. 2023-10-06T01:40:56.804 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.801+0000 7f67a3b73700 1 Processor -- start 2023-10-06T01:40:56.804 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.801+0000 7f67a3b73700 1 -- start start 2023-10-06T01:40:56.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.801+0000 7f67a3b73700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f679c0b3fd0 0x7f679c1431f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:56.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.801+0000 7f67a3b73700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f679c159500 0x7f679c068b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:56.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.801+0000 7f67a3b73700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f679c15dab0 0x7f679c0690a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:56.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.801+0000 7f67a3b73700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c1480f0 con 0x7f679c15dab0 2023-10-06T01:40:56.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.801+0000 7f67a3b73700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c148f50 con 0x7f679c159500 2023-10-06T01:40:56.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.801+0000 7f67a3b73700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f679c148340 con 0x7f679c0b3fd0 2023-10-06T01:40:56.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.801+0000 7f67a190f700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f679c0b3fd0 0x7f679c1431f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:56.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.801+0000 7f67a190f700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f679c0b3fd0 0x7f679c1431f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.124:43440/0 (socket says 172.21.15.124:43440) 2023-10-06T01:40:56.807 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.801+0000 7f67a190f700 1 -- 172.21.15.124:0/3228143239 learned_addr learned my addr 172.21.15.124:0/3228143239 (peer_addr_for_me v2:172.21.15.124:0/0) 2023-10-06T01:40:56.808 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.801+0000 7f67a110e700 1 --2- 172.21.15.124:0/3228143239 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f679c159500 0x7f679c068b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:56.808 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.801+0000 7f67a2110700 1 --2- 172.21.15.124:0/3228143239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f679c15dab0 0x7f679c0690a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:56.808 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.802+0000 7f67a190f700 1 -- 172.21.15.124:0/3228143239 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f679c159500 msgr2=0x7f679c068b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:56.808 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.802+0000 7f67a190f700 1 --2- 172.21.15.124:0/3228143239 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f679c159500 0x7f679c068b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.808 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.802+0000 7f67a190f700 1 -- 172.21.15.124:0/3228143239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f679c15dab0 msgr2=0x7f679c0690a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:56.808 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.802+0000 7f67a190f700 1 --2- 172.21.15.124:0/3228143239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f679c15dab0 0x7f679c0690a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.808 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.802+0000 7f67a190f700 1 -- 172.21.15.124:0/3228143239 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6794009ae0 con 0x7f679c0b3fd0 2023-10-06T01:40:56.808 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.802+0000 7f67a110e700 1 --2- 172.21.15.124:0/3228143239 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f679c159500 0x7f679c068b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:40:56.808 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.802+0000 7f67a2110700 1 --2- 172.21.15.124:0/3228143239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f679c15dab0 0x7f679c0690a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:40:56.808 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.802+0000 7f67a190f700 1 --2- 172.21.15.124:0/3228143239 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f679c0b3fd0 0x7f679c1431f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f679800ec80 tx=0x7f67980091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:40:56.808 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.802+0000 7f6791ffb700 1 -- 172.21.15.124:0/3228143239 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679800c850 con 0x7f679c0b3fd0 2023-10-06T01:40:56.809 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.802+0000 7f6791ffb700 1 -- 172.21.15.124:0/3228143239 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f679800c9b0 con 0x7f679c0b3fd0 2023-10-06T01:40:56.809 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.802+0000 7f67a3b73700 1 -- 172.21.15.124:0/3228143239 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f679c148620 con 0x7f679c0b3fd0 2023-10-06T01:40:56.809 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.802+0000 7f6791ffb700 1 -- 172.21.15.124:0/3228143239 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679800cd50 con 0x7f679c0b3fd0 2023-10-06T01:40:56.809 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.802+0000 7f67a3b73700 1 -- 172.21.15.124:0/3228143239 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f679c148be0 con 0x7f679c0b3fd0 2023-10-06T01:40:56.809 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.802+0000 7f6791ffb700 1 -- 172.21.15.124:0/3228143239 <== mon.2 v2:172.21.15.124:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f6798005c80 con 0x7f679c0b3fd0 2023-10-06T01:40:56.809 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.802+0000 7f6791ffb700 1 -- 172.21.15.124:0/3228143239 <== mon.2 v2:172.21.15.124:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1526+0+0 (secure 0 0 0) 0x7f6798006690 con 0x7f679c0b3fd0 2023-10-06T01:40:56.809 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.803+0000 7f67a3b73700 1 -- 172.21.15.124:0/3228143239 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f679c064050 con 0x7f679c0b3fd0 2023-10-06T01:40:56.809 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.805+0000 7f6791ffb700 1 -- 172.21.15.124:0/3228143239 <== mon.2 v2:172.21.15.124:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f6798021060 con 0x7f679c0b3fd0 2023-10-06T01:40:56.889 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.888+0000 7f67a3b73700 1 -- 172.21.15.124:0/3228143239 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_command({"prefix": "osd new", "uuid": "ae463a90-a9d3-498f-86fa-47a35b3fe889", "id": 7} v 0) v1 -- 0x7f679c0b4470 con 0x7f679c0b3fd0 2023-10-06T01:40:56.897 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.896+0000 7f6791ffb700 1 -- 172.21.15.124:0/3228143239 <== mon.2 v2:172.21.15.124:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ae463a90-a9d3-498f-86fa-47a35b3fe889", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6798017070 con 0x7f679c0b3fd0 2023-10-06T01:40:56.897 INFO:teuthology.orchestra.run.smithi124.stdout:7 2023-10-06T01:40:56.898 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.897+0000 7f67a3b73700 1 -- 172.21.15.124:0/3228143239 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f679c0b3fd0 msgr2=0x7f679c1431f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:56.898 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.897+0000 7f67a3b73700 1 --2- 172.21.15.124:0/3228143239 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f679c0b3fd0 0x7f679c1431f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f679800ec80 tx=0x7f67980091a0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.898 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.897+0000 7f67a3b73700 1 -- 172.21.15.124:0/3228143239 shutdown_connections 2023-10-06T01:40:56.899 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.897+0000 7f67a3b73700 1 --2- 172.21.15.124:0/3228143239 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f679c0b3fd0 0x7f679c1431f0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.899 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.897+0000 7f67a3b73700 1 --2- 172.21.15.124:0/3228143239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f679c15dab0 0x7f679c0690a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.899 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.897+0000 7f67a3b73700 1 --2- 172.21.15.124:0/3228143239 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f679c159500 0x7f679c068b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:56.899 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.897+0000 7f67a3b73700 1 -- 172.21.15.124:0/3228143239 >> 172.21.15.124:0/3228143239 conn(0x7f679c067630 msgr2=0x7f679c0b1d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:40:56.899 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.897+0000 7f67a3b73700 1 -- 172.21.15.124:0/3228143239 shutdown_connections 2023-10-06T01:40:56.899 INFO:teuthology.orchestra.run.smithi124.stderr:2023-10-06T01:40:56.897+0000 7f67a3b73700 1 -- 172.21.15.124:0/3228143239 wait complete. 2023-10-06T01:40:56.910 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-06T01:40:56.910 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-06T01:40:56.913 INFO:tasks.ceph.osd.0:Started 2023-10-06T01:40:56.913 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-06T01:40:56.914 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-06T01:40:56.916 INFO:tasks.ceph.osd.1:Started 2023-10-06T01:40:56.916 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-06T01:40:56.916 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-06T01:40:56.918 INFO:tasks.ceph.osd.2:Started 2023-10-06T01:40:56.918 INFO:tasks.ceph.osd.3:Restarting daemon 2023-10-06T01:40:56.918 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-10-06T01:40:56.920 INFO:tasks.ceph.osd.3:Started 2023-10-06T01:40:56.920 INFO:tasks.ceph.osd.4:Restarting daemon 2023-10-06T01:40:56.920 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-10-06T01:40:56.923 INFO:tasks.ceph.osd.4:Started 2023-10-06T01:40:56.923 INFO:tasks.ceph.osd.5:Restarting daemon 2023-10-06T01:40:56.923 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-10-06T01:40:56.925 INFO:tasks.ceph.osd.5:Started 2023-10-06T01:40:56.926 INFO:tasks.ceph.osd.6:Restarting daemon 2023-10-06T01:40:56.926 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-10-06T01:40:56.928 INFO:tasks.ceph.osd.6:Started 2023-10-06T01:40:56.929 INFO:tasks.ceph.osd.7:Restarting daemon 2023-10-06T01:40:56.929 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-10-06T01:40:56.931 INFO:tasks.ceph.osd.7:Started 2023-10-06T01:40:56.931 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T01:40:57.074 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.073+0000 7fe34736f700 1 Processor -- start 2023-10-06T01:40:57.075 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.073+0000 7fe34736f700 1 -- start start 2023-10-06T01:40:57.077 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.074+0000 7fe34736f700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe34015d560 0x7fe3400d0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:57.077 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.074+0000 7fe34736f700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe34015d930 0x7fe3400b1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:57.077 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.074+0000 7fe34736f700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3400b2ad0 con 0x7fe3401591b0 2023-10-06T01:40:57.077 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.074+0000 7fe34736f700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3400b4160 con 0x7fe34015d560 2023-10-06T01:40:57.077 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.074+0000 7fe34736f700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3400b1f80 con 0x7fe34015d930 2023-10-06T01:40:57.077 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.074+0000 7fe34510b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe34015d560 0x7fe3400d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:57.077 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.074+0000 7fe34510b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe34015d560 0x7fe3400d0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49914/0 (socket says 172.21.15.27:49914) 2023-10-06T01:40:57.077 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.074+0000 7fe34590c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe3401591b0 0x7fe340159630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:33220/0 (socket says 172.21.15.27:33220) 2023-10-06T01:40:57.077 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.074+0000 7fe34510b700 1 -- 172.21.15.27:0/1675728456 learned_addr learned my addr 172.21.15.27:0/1675728456 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:40:57.078 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.074+0000 7fe34490a700 1 --2- 172.21.15.27:0/1675728456 >> v2:172.21.15.124:3300/0 conn(0x7fe34015d930 0x7fe3400b1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:57.078 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.075+0000 7fe34510b700 1 -- 172.21.15.27:0/1675728456 >> v2:172.21.15.124:3300/0 conn(0x7fe34015d930 msgr2=0x7fe3400b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:40:57.078 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.075+0000 7fe34510b700 1 --2- 172.21.15.27:0/1675728456 >> v2:172.21.15.124:3300/0 conn(0x7fe34015d930 0x7fe3400b1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:57.078 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.075+0000 7fe34510b700 1 -- 172.21.15.27:0/1675728456 >> v1:172.21.15.27:6789/0 conn(0x7fe3401591b0 legacy=0x7fe340159630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:57.078 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.075+0000 7fe337fff700 1 -- 172.21.15.27:0/1675728456 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 92510944 0 0) 0x7fe3400b2ad0 con 0x7fe3401591b0 2023-10-06T01:40:57.078 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.075+0000 7fe34510b700 1 -- 172.21.15.27:0/1675728456 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3400b2230 con 0x7fe34015d560 2023-10-06T01:40:57.078 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.075+0000 7fe34510b700 1 --2- 172.21.15.27:0/1675728456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe34015d560 0x7fe3400d0030 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe34015e440 tx=0x7fe330009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7843b9cf6baaa816 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:40:57.078 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.075+0000 7fe337fff700 1 -- 172.21.15.27:0/1675728456 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe330009e70 con 0x7fe34015d560 2023-10-06T01:40:57.078 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.075+0000 7fe337fff700 1 -- 172.21.15.27:0/1675728456 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe33000b860 con 0x7fe34015d560 2023-10-06T01:40:57.078 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.075+0000 7fe337fff700 1 -- 172.21.15.27:0/1675728456 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33000bc00 con 0x7fe34015d560 2023-10-06T01:40:57.078 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.075+0000 7fe34736f700 1 -- 172.21.15.27:0/1675728456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe34015d560 msgr2=0x7fe3400d0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:57.079 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.075+0000 7fe34736f700 1 --2- 172.21.15.27:0/1675728456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe34015d560 0x7fe3400d0030 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe34015e440 tx=0x7fe330009250 comp rx=0 tx=0).stop 2023-10-06T01:40:57.079 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.076+0000 7fe34736f700 1 -- 172.21.15.27:0/1675728456 shutdown_connections 2023-10-06T01:40:57.079 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.076+0000 7fe34736f700 1 --2- 172.21.15.27:0/1675728456 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe34015d560 0x7fe3400d0030 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:57.079 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.076+0000 7fe34736f700 1 --2- 172.21.15.27:0/1675728456 >> v2:172.21.15.124:3300/0 conn(0x7fe34015d930 0x7fe3400b1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:57.079 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.076+0000 7fe34736f700 1 -- 172.21.15.27:0/1675728456 >> 172.21.15.27:0/1675728456 conn(0x7fe3400d1180 msgr2=0x7fe340069640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:40:57.079 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.076+0000 7fe34736f700 1 -- 172.21.15.27:0/1675728456 shutdown_connections 2023-10-06T01:40:57.079 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.076+0000 7fe34736f700 1 -- 172.21.15.27:0/1675728456 wait complete. 2023-10-06T01:40:57.081 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.077+0000 7fe34736f700 1 Processor -- start 2023-10-06T01:40:57.081 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.077+0000 7fe34736f700 1 -- start start 2023-10-06T01:40:57.082 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.078+0000 7fe34736f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3401591b0 0x7fe340156020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:57.082 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.078+0000 7fe34736f700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe34015d560 0x7fe34014e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:57.082 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.078+0000 7fe34736f700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe34015d930 0x7fe34014f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:57.082 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.078+0000 7fe34736f700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe340155310 con 0x7fe3401591b0 2023-10-06T01:40:57.082 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.078+0000 7fe34736f700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe340156560 con 0x7fe34015d930 2023-10-06T01:40:57.082 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.078+0000 7fe34736f700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe340154700 con 0x7fe34015d560 2023-10-06T01:40:57.082 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.078+0000 7fe34510b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3401591b0 0x7fe340156020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:57.082 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.078+0000 7fe34510b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3401591b0 0x7fe340156020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49916/0 (socket says 172.21.15.27:49916) 2023-10-06T01:40:57.082 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.078+0000 7fe34510b700 1 -- 172.21.15.27:0/802372111 learned_addr learned my addr 172.21.15.27:0/802372111 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:40:57.082 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.078+0000 7fe34590c700 1 --2- 172.21.15.27:0/802372111 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe34015d930 0x7fe34014f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:57.083 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.078+0000 7fe34490a700 1 --2- 172.21.15.27:0/802372111 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe34015d560 0x7fe34014e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:57.083 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.078+0000 7fe34510b700 1 -- 172.21.15.27:0/802372111 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe34015d560 msgr2=0x7fe34014e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:57.083 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.078+0000 7fe34510b700 1 --2- 172.21.15.27:0/802372111 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe34015d560 0x7fe34014e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:57.083 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.078+0000 7fe34510b700 1 -- 172.21.15.27:0/802372111 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe34015d930 msgr2=0x7fe34014f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:57.083 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.078+0000 7fe34510b700 1 --2- 172.21.15.27:0/802372111 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe34015d930 0x7fe34014f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:57.083 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.078+0000 7fe34510b700 1 -- 172.21.15.27:0/802372111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3300094f0 con 0x7fe3401591b0 2023-10-06T01:40:57.083 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.078+0000 7fe34590c700 1 --2- 172.21.15.27:0/802372111 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe34015d930 0x7fe34014f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:40:57.083 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.078+0000 7fe34510b700 1 --2- 172.21.15.27:0/802372111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3401591b0 0x7fe340156020 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe340064a20 tx=0x7fe330017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:40:57.083 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.078+0000 7fe3357fa700 1 -- 172.21.15.27:0/802372111 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33000bbe0 con 0x7fe3401591b0 2023-10-06T01:40:57.083 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.078+0000 7fe3357fa700 1 -- 172.21.15.27:0/802372111 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe330000c90 con 0x7fe3401591b0 2023-10-06T01:40:57.083 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.078+0000 7fe34736f700 1 -- 172.21.15.27:0/802372111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3401549e0 con 0x7fe3401591b0 2023-10-06T01:40:57.083 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.078+0000 7fe3357fa700 1 -- 172.21.15.27:0/802372111 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe330011650 con 0x7fe3401591b0 2023-10-06T01:40:57.084 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.079+0000 7fe34736f700 1 -- 172.21.15.27:0/802372111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe340154fa0 con 0x7fe3401591b0 2023-10-06T01:40:57.084 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.079+0000 7fe3357fa700 1 -- 172.21.15.27:0/802372111 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fe330011cc0 con 0x7fe3401591b0 2023-10-06T01:40:57.084 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.079+0000 7fe3357fa700 1 -- 172.21.15.27:0/802372111 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7fe3300253f0 con 0x7fe3401591b0 2023-10-06T01:40:57.084 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.080+0000 7fe34736f700 1 -- 172.21.15.27:0/802372111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe340064050 con 0x7fe3401591b0 2023-10-06T01:40:57.084 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.083+0000 7fe3357fa700 1 -- 172.21.15.27:0/802372111 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fe330033060 con 0x7fe3401591b0 2023-10-06T01:40:57.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.154+0000 7fe34736f700 1 -- 172.21.15.27:0/802372111 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3400b2510 con 0x7fe3401591b0 2023-10-06T01:40:57.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.154+0000 7fe3357fa700 1 -- 172.21.15.27:0/802372111 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7fe330015020 con 0x7fe3401591b0 2023-10-06T01:40:57.156 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:40:57.156 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":10,"fsid":"20a7bff0-74f8-4f2f-b6c6-899e804bc965","created":"2023-10-06T01:40:54.474215+0000","modified":"2023-10-06T01:40:56.890960+0000","last_up_change":"0.000000","last_in_change":"2023-10-06T01:40:56.890960+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"b902c86d-800c-40c5-b152-b545e50d02c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5b4a70e5-b06e-48cd-bfc4-7962c5d05386","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"22d9de46-4073-4336-9122-5227e601eca0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"438bb2d3-34e2-4b7c-9734-8360c1b67650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1956a8e-e0f8-4e7c-acbb-40037415a354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"306c59d4-846d-4b73-b1c9-051bec9e9368","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"15463e53-ac8d-47d1-b4f6-c0d31f29e4a9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae463a90-a9d3-498f-86fa-47a35b3fe889","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T01:40:57.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.155+0000 7fe34736f700 1 -- 172.21.15.27:0/802372111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3401591b0 msgr2=0x7fe340156020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:57.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.155+0000 7fe34736f700 1 --2- 172.21.15.27:0/802372111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3401591b0 0x7fe340156020 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe340064a20 tx=0x7fe330017000 comp rx=0 tx=0).stop 2023-10-06T01:40:57.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.155+0000 7fe34736f700 1 -- 172.21.15.27:0/802372111 shutdown_connections 2023-10-06T01:40:57.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.155+0000 7fe34736f700 1 --2- 172.21.15.27:0/802372111 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe34015d930 0x7fe34014f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:57.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.155+0000 7fe34736f700 1 --2- 172.21.15.27:0/802372111 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3401591b0 0x7fe340156020 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:57.158 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.155+0000 7fe34736f700 1 --2- 172.21.15.27:0/802372111 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe34015d560 0x7fe34014e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:57.158 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.155+0000 7fe34736f700 1 -- 172.21.15.27:0/802372111 >> 172.21.15.27:0/802372111 conn(0x7fe3400d1180 msgr2=0x7fe340157910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:40:57.158 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.156+0000 7fe34736f700 1 -- 172.21.15.27:0/802372111 shutdown_connections 2023-10-06T01:40:57.158 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:40:57.156+0000 7fe34736f700 1 -- 172.21.15.27:0/802372111 wait complete. 2023-10-06T01:40:57.168 INFO:tasks.ceph.mgr.x.smithi124.stderr:2023-10-06T01:40:57.167+0000 7fdf65db9180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-06T01:40:57.169 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-06T01:40:57.169 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-06T01:40:57.170 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-10-06T01:40:57.168+0000 7fc8049c8180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-06T01:40:57.245 INFO:tasks.ceph.mgr.x.smithi124.stderr:2023-10-06T01:40:57.243+0000 7fdf65db9180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-06T01:40:57.250 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-10-06T01:40:57.249+0000 7fc8049c8180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-06T01:40:57.276 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-06T01:40:57.274+0000 7f788dc0d540 -1 Falling back to public interface 2023-10-06T01:40:57.324 INFO:tasks.ceph.mgr.x.smithi124.stderr:2023-10-06T01:40:57.323+0000 7fdf65db9180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-06T01:40:57.331 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-10-06T01:40:57.330+0000 7fc8049c8180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-06T01:40:57.520 INFO:tasks.ceph.osd.4.smithi124.stderr:2023-10-06T01:40:57.519+0000 7f9542e39540 -1 Falling back to public interface 2023-10-06T01:40:57.524 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-06T01:40:57.522+0000 7f849d42c540 -1 Falling back to public interface 2023-10-06T01:40:57.524 INFO:tasks.ceph.osd.6.smithi124.stderr:2023-10-06T01:40:57.523+0000 7f0c9b9c9540 -1 Falling back to public interface 2023-10-06T01:40:57.527 INFO:tasks.ceph.osd.2.smithi027.stderr:2023-10-06T01:40:57.526+0000 7ffa4a61d540 -1 Falling back to public interface 2023-10-06T01:40:57.534 INFO:tasks.ceph.osd.5.smithi124.stderr:2023-10-06T01:40:57.533+0000 7ff7eaacf540 -1 Falling back to public interface 2023-10-06T01:40:57.552 INFO:tasks.ceph.osd.3.smithi027.stderr:2023-10-06T01:40:57.551+0000 7f70b205a540 -1 Falling back to public interface 2023-10-06T01:40:57.561 INFO:tasks.ceph.osd.7.smithi124.stderr:2023-10-06T01:40:57.559+0000 7f31a1bc5540 -1 Falling back to public interface 2023-10-06T01:40:57.905 INFO:tasks.ceph.mgr.x.smithi124.stderr:2023-10-06T01:40:57.904+0000 7fdf65db9180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-06T01:40:57.943 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-10-06T01:40:57.942+0000 7fc8049c8180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-06T01:40:57.974 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-06T01:40:58.041 INFO:tasks.ceph.mgr.x.smithi124.stderr:2023-10-06T01:40:58.040+0000 7fdf65db9180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-06T01:40:58.084 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.083+0000 7f9a8a2ea700 1 Processor -- start 2023-10-06T01:40:58.084 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.083+0000 7f9a8a2ea700 1 -- start start 2023-10-06T01:40:58.086 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.084+0000 7f9a8a2ea700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9a84159b60 0x7f9a841645a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:58.086 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.084+0000 7f9a8a2ea700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a840cfeb0 con 0x7f9a840caeb0 2023-10-06T01:40:58.086 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.084+0000 7f9a8a2ea700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a840cf2a0 con 0x7f9a840cb280 2023-10-06T01:40:58.086 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.084+0000 7f9a8a2ea700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a840cf580 con 0x7f9a84159b60 2023-10-06T01:40:58.086 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.084+0000 7f9a88887700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9a84159b60 0x7f9a841645a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:58.086 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.084+0000 7f9a83fff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f9a840caeb0 0x7f9a840d0dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:33224/0 (socket says 172.21.15.27:33224) 2023-10-06T01:40:58.086 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.084+0000 7f9a83fff700 1 -- 172.21.15.27:0/4208683097 learned_addr learned my addr 172.21.15.27:0/4208683097 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:40:58.087 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.084+0000 7f9a88887700 1 -- 172.21.15.27:0/4208683097 >> v1:172.21.15.124:6789/0 conn(0x7f9a840cb280 legacy=0x7f9a84159450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:58.087 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.084+0000 7f9a88887700 1 -- 172.21.15.27:0/4208683097 >> v1:172.21.15.27:6789/0 conn(0x7f9a840caeb0 legacy=0x7f9a840d0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:58.087 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.084+0000 7f9a88887700 1 -- 172.21.15.27:0/4208683097 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a840cf860 con 0x7f9a84159b60 2023-10-06T01:40:58.087 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.084+0000 7f9a88887700 1 --2- 172.21.15.27:0/4208683097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a84159b60 0x7f9a841645a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c009e60 tx=0x7f9a7c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b96e778b2143f1e server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:40:58.087 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.084+0000 7f9a82ffd700 1 -- 172.21.15.27:0/4208683097 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c00e070 con 0x7f9a84159b60 2023-10-06T01:40:58.087 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.084+0000 7f9a82ffd700 1 -- 172.21.15.27:0/4208683097 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a7c00bc30 con 0x7f9a84159b60 2023-10-06T01:40:58.087 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.084+0000 7f9a82ffd700 1 -- 172.21.15.27:0/4208683097 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c012430 con 0x7f9a84159b60 2023-10-06T01:40:58.090 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.085+0000 7f9a8a2ea700 1 -- 172.21.15.27:0/4208683097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a84159b60 msgr2=0x7f9a841645a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:58.090 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.085+0000 7f9a8a2ea700 1 --2- 172.21.15.27:0/4208683097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a84159b60 0x7f9a841645a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c009e60 tx=0x7f9a7c009780 comp rx=0 tx=0).stop 2023-10-06T01:40:58.090 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.085+0000 7f9a8a2ea700 1 -- 172.21.15.27:0/4208683097 shutdown_connections 2023-10-06T01:40:58.090 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.085+0000 7f9a8a2ea700 1 --2- 172.21.15.27:0/4208683097 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a84159b60 0x7f9a841645a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:58.090 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.085+0000 7f9a8a2ea700 1 -- 172.21.15.27:0/4208683097 >> 172.21.15.27:0/4208683097 conn(0x7f9a8406ce10 msgr2=0x7f9a840d06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:40:58.090 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.085+0000 7f9a8a2ea700 1 -- 172.21.15.27:0/4208683097 shutdown_connections 2023-10-06T01:40:58.091 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.085+0000 7f9a8a2ea700 1 -- 172.21.15.27:0/4208683097 wait complete. 2023-10-06T01:40:58.091 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.086+0000 7f9a8a2ea700 1 Processor -- start 2023-10-06T01:40:58.091 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.086+0000 7f9a8a2ea700 1 -- start start 2023-10-06T01:40:58.091 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.087+0000 7f9a8a2ea700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9a840caeb0 0x7f9a840c6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:58.091 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.087+0000 7f9a8a2ea700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9a840cb280 0x7f9a840ae830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:58.091 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.087+0000 7f9a8a2ea700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a84159b60 0x7f9a840aed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:40:58.091 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.087+0000 7f9a8a2ea700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8406f540 con 0x7f9a84159b60 2023-10-06T01:40:58.091 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.087+0000 7f9a8a2ea700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a840c72a0 con 0x7f9a840cb280 2023-10-06T01:40:58.092 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.087+0000 7f9a8a2ea700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8406e930 con 0x7f9a840caeb0 2023-10-06T01:40:58.092 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.087+0000 7f9a88887700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a84159b60 0x7f9a840aed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:58.092 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.087+0000 7f9a88887700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a84159b60 0x7f9a840aed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49928/0 (socket says 172.21.15.27:49928) 2023-10-06T01:40:58.092 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.087+0000 7f9a88887700 1 -- 172.21.15.27:0/2558502247 learned_addr learned my addr 172.21.15.27:0/2558502247 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:40:58.092 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.087+0000 7f9a83fff700 1 --2- 172.21.15.27:0/2558502247 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9a840caeb0 0x7f9a840c6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:58.092 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.087+0000 7f9a837fe700 1 --2- 172.21.15.27:0/2558502247 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9a840cb280 0x7f9a840ae830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:40:58.092 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.087+0000 7f9a88887700 1 -- 172.21.15.27:0/2558502247 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9a840caeb0 msgr2=0x7f9a840c6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:58.092 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.087+0000 7f9a88887700 1 --2- 172.21.15.27:0/2558502247 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9a840caeb0 0x7f9a840c6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:58.092 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.087+0000 7f9a88887700 1 -- 172.21.15.27:0/2558502247 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9a840cb280 msgr2=0x7f9a840ae830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:58.093 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.087+0000 7f9a88887700 1 --2- 172.21.15.27:0/2558502247 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9a840cb280 0x7f9a840ae830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:58.093 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.087+0000 7f9a88887700 1 -- 172.21.15.27:0/2558502247 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a7c009a20 con 0x7f9a84159b60 2023-10-06T01:40:58.093 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.087+0000 7f9a88887700 1 --2- 172.21.15.27:0/2558502247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a84159b60 0x7f9a840aed70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c0097c0 tx=0x7f9a7c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:40:58.093 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.088+0000 7f9a6bfff700 1 -- 172.21.15.27:0/2558502247 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c012e10 con 0x7f9a84159b60 2023-10-06T01:40:58.093 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.088+0000 7f9a6bfff700 1 -- 172.21.15.27:0/2558502247 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a7c01e630 con 0x7f9a84159b60 2023-10-06T01:40:58.093 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.088+0000 7f9a8a2ea700 1 -- 172.21.15.27:0/2558502247 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a8406ec10 con 0x7f9a84159b60 2023-10-06T01:40:58.093 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.088+0000 7f9a6bfff700 1 -- 172.21.15.27:0/2558502247 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7c01e9f0 con 0x7f9a84159b60 2023-10-06T01:40:58.093 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.088+0000 7f9a8a2ea700 1 -- 172.21.15.27:0/2558502247 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a8406f1d0 con 0x7f9a84159b60 2023-10-06T01:40:58.094 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.088+0000 7f9a6bfff700 1 -- 172.21.15.27:0/2558502247 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f9a7c024070 con 0x7f9a84159b60 2023-10-06T01:40:58.094 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.088+0000 7f9a6bfff700 1 -- 172.21.15.27:0/2558502247 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7f9a7c027e20 con 0x7f9a84159b60 2023-10-06T01:40:58.094 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-10-06T01:40:58.088+0000 7fc8049c8180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-06T01:40:58.095 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.089+0000 7f9a8a2ea700 1 -- 172.21.15.27:0/2558502247 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a840cfb40 con 0x7f9a84159b60 2023-10-06T01:40:58.095 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.091+0000 7f9a6bfff700 1 -- 172.21.15.27:0/2558502247 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f9a7c01ee20 con 0x7f9a84159b60 2023-10-06T01:40:58.132 INFO:tasks.ceph.mgr.x.smithi124.stderr:2023-10-06T01:40:58.131+0000 7fdf65db9180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-06T01:40:58.179 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.178+0000 7f9a8a2ea700 1 -- 172.21.15.27:0/2558502247 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a84064120 con 0x7f9a84159b60 2023-10-06T01:40:58.180 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.179+0000 7f9a6bfff700 1 -- 172.21.15.27:0/2558502247 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7f9a7c027740 con 0x7f9a84159b60 2023-10-06T01:40:58.180 INFO:teuthology.misc.health.smithi027.stdout: 2023-10-06T01:40:58.180 INFO:teuthology.misc.health.smithi027.stdout:{"epoch":10,"fsid":"20a7bff0-74f8-4f2f-b6c6-899e804bc965","created":"2023-10-06T01:40:54.474215+0000","modified":"2023-10-06T01:40:56.890960+0000","last_up_change":"0.000000","last_in_change":"2023-10-06T01:40:56.890960+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"b902c86d-800c-40c5-b152-b545e50d02c4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5b4a70e5-b06e-48cd-bfc4-7962c5d05386","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"22d9de46-4073-4336-9122-5227e601eca0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"438bb2d3-34e2-4b7c-9734-8360c1b67650","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"e1956a8e-e0f8-4e7c-acbb-40037415a354","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"306c59d4-846d-4b73-b1c9-051bec9e9368","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"15463e53-ac8d-47d1-b4f6-c0d31f29e4a9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ae463a90-a9d3-498f-86fa-47a35b3fe889","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T01:40:58.181 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.179+0000 7f9a8a2ea700 1 -- 172.21.15.27:0/2558502247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a84159b60 msgr2=0x7f9a840aed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:40:58.181 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.179+0000 7f9a8a2ea700 1 --2- 172.21.15.27:0/2558502247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a84159b60 0x7f9a840aed70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9a7c0097c0 tx=0x7f9a7c019000 comp rx=0 tx=0).stop 2023-10-06T01:40:58.181 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.179+0000 7f9a8a2ea700 1 -- 172.21.15.27:0/2558502247 shutdown_connections 2023-10-06T01:40:58.181 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.179+0000 7f9a8a2ea700 1 --2- 172.21.15.27:0/2558502247 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9a840caeb0 0x7f9a840c6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:58.181 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.179+0000 7f9a8a2ea700 1 --2- 172.21.15.27:0/2558502247 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9a84159b60 0x7f9a840aed70 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:58.181 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.179+0000 7f9a8a2ea700 1 --2- 172.21.15.27:0/2558502247 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9a840cb280 0x7f9a840ae830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:40:58.182 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.179+0000 7f9a8a2ea700 1 -- 172.21.15.27:0/2558502247 >> 172.21.15.27:0/2558502247 conn(0x7f9a8406ce10 msgr2=0x7f9a840d0340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:40:58.182 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.180+0000 7f9a8a2ea700 1 -- 172.21.15.27:0/2558502247 shutdown_connections 2023-10-06T01:40:58.182 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:40:58.180+0000 7f9a8a2ea700 1 -- 172.21.15.27:0/2558502247 wait complete. 2023-10-06T01:40:58.191 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-06T01:40:58.199 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-10-06T01:40:58.198+0000 7fc8049c8180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-06T01:40:58.378 INFO:tasks.ceph.mgr.x.smithi124.stderr:2023-10-06T01:40:58.377+0000 7fdf65db9180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-06T01:40:58.455 INFO:tasks.ceph.mgr.x.smithi124.stderr:2023-10-06T01:40:58.454+0000 7fdf65db9180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-06T01:40:58.457 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-10-06T01:40:58.456+0000 7fc8049c8180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-06T01:40:58.537 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-10-06T01:40:58.536+0000 7fc8049c8180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-06T01:40:59.274 INFO:tasks.ceph.mgr.x.smithi124.stderr:2023-10-06T01:40:59.272+0000 7fdf65db9180 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-06T01:40:59.346 INFO:tasks.ceph.mgr.x.smithi124.stderr:2023-10-06T01:40:59.345+0000 7fdf65db9180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-06T01:40:59.390 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-10-06T01:40:59.389+0000 7fc8049c8180 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-06T01:40:59.419 INFO:tasks.ceph.mgr.x.smithi124.stderr:2023-10-06T01:40:59.418+0000 7fdf65db9180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-06T01:40:59.467 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-10-06T01:40:59.466+0000 7fc8049c8180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-06T01:40:59.543 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-10-06T01:40:59.542+0000 7fc8049c8180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-06T01:40:59.552 INFO:tasks.ceph.mgr.x.smithi124.stderr:2023-10-06T01:40:59.551+0000 7fdf65db9180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-06T01:40:59.635 INFO:tasks.ceph.mgr.x.smithi124.stderr:2023-10-06T01:40:59.633+0000 7fdf65db9180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-06T01:40:59.681 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-10-06T01:40:59.680+0000 7fc8049c8180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-06T01:40:59.770 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-10-06T01:40:59.769+0000 7fc8049c8180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-06T01:40:59.786 INFO:tasks.ceph.mgr.x.smithi124.stderr:2023-10-06T01:40:59.785+0000 7fdf65db9180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-06T01:40:59.862 INFO:tasks.ceph.mgr.x.smithi124.stderr:2023-10-06T01:40:59.860+0000 7fdf65db9180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-06T01:40:59.930 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-10-06T01:40:59.929+0000 7fc8049c8180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-06T01:40:59.935 INFO:tasks.ceph.mgr.x.smithi124.stderr:2023-10-06T01:40:59.934+0000 7fdf65db9180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-06T01:41:00.014 INFO:tasks.ceph.mgr.x.smithi124.stderr:2023-10-06T01:41:00.013+0000 7fdf65db9180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-06T01:41:00.016 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-10-06T01:41:00.015+0000 7fc8049c8180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-06T01:41:00.116 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-10-06T01:41:00.115+0000 7fc8049c8180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-06T01:41:00.197 INFO:tasks.ceph.mgr.y.smithi027.stderr:2023-10-06T01:41:00.196+0000 7fc8049c8180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-06T01:41:00.820 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-06T01:41:00.818+0000 7f849d42c540 -1 osd.1 0 log_to_monitors true 2023-10-06T01:41:00.822 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-06T01:41:00.821+0000 7f788dc0d540 -1 osd.0 0 log_to_monitors true 2023-10-06T01:41:00.994 INFO:tasks.ceph.osd.4.smithi124.stderr:2023-10-06T01:41:00.993+0000 7f9542e39540 -1 osd.4 0 log_to_monitors true 2023-10-06T01:41:01.013 INFO:tasks.ceph.osd.2.smithi027.stderr:2023-10-06T01:41:01.012+0000 7ffa4a61d540 -1 osd.2 0 log_to_monitors true 2023-10-06T01:41:01.040 INFO:tasks.ceph.mgr.x.smithi124.stderr:2023-10-06T01:41:01.039+0000 7fdf1f1bd700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-10-06T01:41:01.040 INFO:tasks.ceph.mgr.x.smithi124.stderr:2023-10-06T01:41:01.039+0000 7fdf1f1bd700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-10-06T01:41:01.048 INFO:tasks.ceph.osd.5.smithi124.stderr:2023-10-06T01:41:01.047+0000 7ff7eaacf540 -1 osd.5 0 log_to_monitors true 2023-10-06T01:41:01.054 INFO:tasks.ceph.osd.6.smithi124.stderr:2023-10-06T01:41:01.053+0000 7f0c9b9c9540 -1 osd.6 0 log_to_monitors true 2023-10-06T01:41:01.081 INFO:tasks.ceph.osd.3.smithi027.stderr:2023-10-06T01:41:01.080+0000 7f70b205a540 -1 osd.3 0 log_to_monitors true 2023-10-06T01:41:01.096 INFO:tasks.ceph.osd.7.smithi124.stderr:2023-10-06T01:41:01.095+0000 7f31a1bc5540 -1 osd.7 0 log_to_monitors true 2023-10-06T01:41:04.236 INFO:tasks.ceph.osd.3.smithi027.stderr:2023-10-06T01:41:04.235+0000 7f70a8c7c700 -1 osd.3 0 waiting for initial osdmap 2023-10-06T01:41:04.249 INFO:tasks.ceph.osd.3.smithi027.stderr:2023-10-06T01:41:04.248+0000 7f70a1a73700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-06T01:41:04.300 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-06T01:41:04.298+0000 7f788482f700 -1 osd.0 0 waiting for initial osdmap 2023-10-06T01:41:04.306 INFO:tasks.ceph.osd.0.smithi027.stderr:2023-10-06T01:41:04.304+0000 7f787d626700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-06T01:41:04.310 INFO:tasks.ceph.osd.6.smithi124.stderr:2023-10-06T01:41:04.310+0000 7f0c925eb700 -1 osd.6 0 waiting for initial osdmap 2023-10-06T01:41:04.320 INFO:tasks.ceph.osd.6.smithi124.stderr:2023-10-06T01:41:04.320+0000 7f0c8b3e2700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-06T01:41:04.322 INFO:tasks.ceph.osd.7.smithi124.stderr:2023-10-06T01:41:04.322+0000 7f3199fea700 -1 osd.7 0 waiting for initial osdmap 2023-10-06T01:41:04.326 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-06T01:41:04.325+0000 7f849404e700 -1 osd.1 0 waiting for initial osdmap 2023-10-06T01:41:04.327 INFO:tasks.ceph.osd.7.smithi124.stderr:2023-10-06T01:41:04.326+0000 7f31915de700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-06T01:41:04.331 INFO:tasks.ceph.osd.1.smithi027.stderr:2023-10-06T01:41:04.330+0000 7f848ce45700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-06T01:41:04.340 INFO:tasks.ceph.osd.2.smithi027.stderr:2023-10-06T01:41:04.339+0000 7ffa4123f700 -1 osd.2 0 waiting for initial osdmap 2023-10-06T01:41:04.342 INFO:tasks.ceph.osd.4.smithi124.stderr:2023-10-06T01:41:04.342+0000 7f9539a5b700 -1 osd.4 0 waiting for initial osdmap 2023-10-06T01:41:04.343 INFO:tasks.ceph.osd.2.smithi027.stderr:2023-10-06T01:41:04.342+0000 7ffa3a036700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-06T01:41:04.351 INFO:tasks.ceph.osd.4.smithi124.stderr:2023-10-06T01:41:04.351+0000 7f9532852700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-06T01:41:04.351 INFO:tasks.ceph.osd.5.smithi124.stderr:2023-10-06T01:41:04.351+0000 7ff7e16f1700 -1 osd.5 0 waiting for initial osdmap 2023-10-06T01:41:04.356 INFO:tasks.ceph.osd.5.smithi124.stderr:2023-10-06T01:41:04.356+0000 7ff7da4e8700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-06T01:41:04.996 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-06T01:41:05.109 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.108+0000 7f006a925700 1 Processor -- start 2023-10-06T01:41:05.109 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.108+0000 7f006a925700 1 -- start start 2023-10-06T01:41:05.112 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.109+0000 7f006a925700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f00640c9900 0x7f006406b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:05.112 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.109+0000 7f006a925700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f00640cd290 0x7f00640cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:05.112 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.109+0000 7f006a925700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00640b0880 con 0x7f00640c9cd0 2023-10-06T01:41:05.112 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.109+0000 7f006a925700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0064156a60 con 0x7f00640c9900 2023-10-06T01:41:05.112 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.109+0000 7f006a925700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00640b1080 con 0x7f00640cd290 2023-10-06T01:41:05.112 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.109+0000 7f0063fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f00640c9900 0x7f006406b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:05.112 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.109+0000 7f0063fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f00640c9900 0x7f006406b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49986/0 (socket says 172.21.15.27:49986) 2023-10-06T01:41:05.112 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.109+0000 7f0063fff700 1 -- 172.21.15.27:0/1786149867 learned_addr learned my addr 172.21.15.27:0/1786149867 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:05.112 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.109+0000 7f0063fff700 1 -- 172.21.15.27:0/1786149867 >> v2:172.21.15.124:3300/0 conn(0x7f00640cd290 msgr2=0x7f00640cd6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:05.113 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.109+0000 7f0063fff700 1 --2- 172.21.15.27:0/1786149867 >> v2:172.21.15.124:3300/0 conn(0x7f00640cd290 0x7f00640cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:05.113 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.109+0000 7f0063fff700 1 -- 172.21.15.27:0/1786149867 >> v1:172.21.15.124:6789/0 conn(0x7f00640c9cd0 legacy=0x7f00640b0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:05.113 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.109+0000 7f0063fff700 1 -- 172.21.15.27:0/1786149867 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00640cdbf0 con 0x7f00640c9900 2023-10-06T01:41:05.113 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.109+0000 7f0063fff700 1 --2- 172.21.15.27:0/1786149867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00640c9900 0x7f006406b720 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f006415b1c0 tx=0x7f0054009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed8767b8733b2a9d server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:05.113 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.109+0000 7f0062ffd700 1 -- 172.21.15.27:0/1786149867 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005400b820 con 0x7f00640c9900 2023-10-06T01:41:05.113 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.109+0000 7f0062ffd700 1 -- 172.21.15.27:0/1786149867 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f005400b980 con 0x7f00640c9900 2023-10-06T01:41:05.113 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.109+0000 7f0062ffd700 1 -- 172.21.15.27:0/1786149867 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005400bd20 con 0x7f00640c9900 2023-10-06T01:41:05.113 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.110+0000 7f006a925700 1 -- 172.21.15.27:0/1786149867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00640c9900 msgr2=0x7f006406b720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:05.113 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.110+0000 7f006a925700 1 --2- 172.21.15.27:0/1786149867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00640c9900 0x7f006406b720 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f006415b1c0 tx=0x7f0054009250 comp rx=0 tx=0).stop 2023-10-06T01:41:05.113 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.110+0000 7f006a925700 1 -- 172.21.15.27:0/1786149867 shutdown_connections 2023-10-06T01:41:05.114 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.110+0000 7f006a925700 1 --2- 172.21.15.27:0/1786149867 >> v2:172.21.15.124:3300/0 conn(0x7f00640cd290 0x7f00640cd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:05.114 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.110+0000 7f006a925700 1 --2- 172.21.15.27:0/1786149867 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00640c9900 0x7f006406b720 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:05.114 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.110+0000 7f006a925700 1 -- 172.21.15.27:0/1786149867 >> 172.21.15.27:0/1786149867 conn(0x7f0064068130 msgr2=0x7f006406d740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:05.114 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.110+0000 7f006a925700 1 -- 172.21.15.27:0/1786149867 shutdown_connections 2023-10-06T01:41:05.114 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.110+0000 7f006a925700 1 -- 172.21.15.27:0/1786149867 wait complete. 2023-10-06T01:41:05.117 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.111+0000 7f006a925700 1 Processor -- start 2023-10-06T01:41:05.117 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.111+0000 7f006a925700 1 -- start start 2023-10-06T01:41:05.117 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.112+0000 7f006a925700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f00640c9900 0x7f0064147ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:05.117 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.112+0000 7f006a925700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f00640c9cd0 0x7f00640c5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:05.117 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.112+0000 7f006a925700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00640cd290 0x7f00640c7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:05.117 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.112+0000 7f006a925700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0064148430 con 0x7f00640cd290 2023-10-06T01:41:05.117 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.112+0000 7f006a925700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0064157840 con 0x7f00640c9900 2023-10-06T01:41:05.118 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.112+0000 7f006a925700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0064157b20 con 0x7f00640c9cd0 2023-10-06T01:41:05.118 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.112+0000 7f0068ec2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00640cd290 0x7f00640c7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:05.118 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.112+0000 7f0068ec2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00640cd290 0x7f00640c7820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:49992/0 (socket says 172.21.15.27:49992) 2023-10-06T01:41:05.118 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.112+0000 7f0068ec2700 1 -- 172.21.15.27:0/926622459 learned_addr learned my addr 172.21.15.27:0/926622459 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:05.118 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.112+0000 7f0063fff700 1 --2- 172.21.15.27:0/926622459 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f00640c9900 0x7f0064147ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:05.118 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.112+0000 7f00637fe700 1 --2- 172.21.15.27:0/926622459 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f00640c9cd0 0x7f00640c5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:05.118 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.112+0000 7f0068ec2700 1 -- 172.21.15.27:0/926622459 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f00640c9cd0 msgr2=0x7f00640c5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:05.118 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.112+0000 7f0068ec2700 1 --2- 172.21.15.27:0/926622459 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f00640c9cd0 0x7f00640c5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:05.118 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.112+0000 7f0068ec2700 1 -- 172.21.15.27:0/926622459 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f00640c9900 msgr2=0x7f0064147ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:05.118 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.112+0000 7f0068ec2700 1 --2- 172.21.15.27:0/926622459 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f00640c9900 0x7f0064147ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:05.119 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.112+0000 7f0068ec2700 1 -- 172.21.15.27:0/926622459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00540094f0 con 0x7f00640cd290 2023-10-06T01:41:05.119 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.112+0000 7f0063fff700 1 --2- 172.21.15.27:0/926622459 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f00640c9900 0x7f0064147ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:05.119 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.113+0000 7f0068ec2700 1 --2- 172.21.15.27:0/926622459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00640cd290 0x7f00640c7820 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f005c00bc60 tx=0x7f005c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:05.119 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.113+0000 7f004bfff700 1 -- 172.21.15.27:0/926622459 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005c0092e0 con 0x7f00640cd290 2023-10-06T01:41:05.119 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.113+0000 7f004bfff700 1 -- 172.21.15.27:0/926622459 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f005c00c7b0 con 0x7f00640cd290 2023-10-06T01:41:05.119 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.113+0000 7f006a925700 1 -- 172.21.15.27:0/926622459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00640c68d0 con 0x7f00640cd290 2023-10-06T01:41:05.119 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.113+0000 7f004bfff700 1 -- 172.21.15.27:0/926622459 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005c00cb90 con 0x7f00640cd290 2023-10-06T01:41:05.119 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.113+0000 7f006a925700 1 -- 172.21.15.27:0/926622459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00640c6e90 con 0x7f00640cd290 2023-10-06T01:41:05.119 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.114+0000 7f006a925700 1 -- 172.21.15.27:0/926622459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0064069d50 con 0x7f00640cd290 2023-10-06T01:41:05.119 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.114+0000 7f004bfff700 1 -- 172.21.15.27:0/926622459 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) v1 ==== 84684+0+0 (secure 0 0 0) 0x7f005c012070 con 0x7f00640cd290 2023-10-06T01:41:05.120 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.114+0000 7f004bfff700 1 --2- 172.21.15.27:0/926622459 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0050066ea0 0x7f0050069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:05.120 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.115+0000 7f004bfff700 1 -- 172.21.15.27:0/926622459 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2227+0+0 (secure 0 0 0) 0x7f005c0832c0 con 0x7f00640cd290 2023-10-06T01:41:05.120 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.115+0000 7f0063fff700 1 --2- 172.21.15.27:0/926622459 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0050066ea0 0x7f0050069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:05.120 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.115+0000 7f0063fff700 1 --2- 172.21.15.27:0/926622459 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0050066ea0 0x7f0050069350 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f006415b1c0 tx=0x7f0054016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:05.120 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.119+0000 7f004bfff700 1 -- 172.21.15.27:0/926622459 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f005c0513a0 con 0x7f00640cd290 2023-10-06T01:41:05.319 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.318+0000 7f006a925700 1 -- 172.21.15.27:0/926622459 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00640b0ac0 con 0x7f00640cd290 2023-10-06T01:41:05.320 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.319+0000 7f004bfff700 1 -- 172.21.15.27:0/926622459 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+10396 (secure 0 0 0) 0x7f005c01a020 con 0x7f00640cd290 2023-10-06T01:41:05.320 INFO:teuthology.misc.health.smithi027.stdout: 2023-10-06T01:41:05.320 INFO:teuthology.misc.health.smithi027.stdout:{"epoch":13,"fsid":"20a7bff0-74f8-4f2f-b6c6-899e804bc965","created":"2023-10-06T01:40:54.474215+0000","modified":"2023-10-06T01:41:05.242847+0000","last_up_change":"2023-10-06T01:41:05.242847+0000","last_in_change":"2023-10-06T01:40:56.890960+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"b902c86d-800c-40c5-b152-b545e50d02c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":90644},{"type":"v1","addr":"172.21.15.27:6801","nonce":90644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":90644},{"type":"v1","addr":"172.21.15.27:6803","nonce":90644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":90644},{"type":"v1","addr":"172.21.15.27:6807","nonce":90644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":90644},{"type":"v1","addr":"172.21.15.27:6805","nonce":90644}]},"public_addr":"172.21.15.27:6801/90644","cluster_addr":"172.21.15.27:6803/90644","heartbeat_back_addr":"172.21.15.27:6807/90644","heartbeat_front_addr":"172.21.15.27:6805/90644","state":["exists","up"]},{"osd":1,"uuid":"5b4a70e5-b06e-48cd-bfc4-7962c5d05386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":90643},{"type":"v1","addr":"172.21.15.27:6809","nonce":90643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":90643},{"type":"v1","addr":"172.21.15.27:6811","nonce":90643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":90643},{"type":"v1","addr":"172.21.15.27:6815","nonce":90643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":90643},{"type":"v1","addr":"172.21.15.27:6813","nonce":90643}]},"public_addr":"172.21.15.27:6809/90643","cluster_addr":"172.21.15.27:6811/90643","heartbeat_back_addr":"172.21.15.27:6815/90643","heartbeat_front_addr":"172.21.15.27:6813/90643","state":["exists","up"]},{"osd":2,"uuid":"22d9de46-4073-4336-9122-5227e601eca0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":90645},{"type":"v1","addr":"172.21.15.27:6817","nonce":90645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":90645},{"type":"v1","addr":"172.21.15.27:6819","nonce":90645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":90645},{"type":"v1","addr":"172.21.15.27:6823","nonce":90645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":90645},{"type":"v1","addr":"172.21.15.27:6821","nonce":90645}]},"public_addr":"172.21.15.27:6817/90645","cluster_addr":"172.21.15.27:6819/90645","heartbeat_back_addr":"172.21.15.27:6823/90645","heartbeat_front_addr":"172.21.15.27:6821/90645","state":["exists","up"]},{"osd":3,"uuid":"438bb2d3-34e2-4b7c-9734-8360c1b67650","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":90668},{"type":"v1","addr":"172.21.15.27:6825","nonce":90668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":90668},{"type":"v1","addr":"172.21.15.27:6827","nonce":90668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":90668},{"type":"v1","addr":"172.21.15.27:6831","nonce":90668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":90668},{"type":"v1","addr":"172.21.15.27:6829","nonce":90668}]},"public_addr":"172.21.15.27:6825/90668","cluster_addr":"172.21.15.27:6827/90668","heartbeat_back_addr":"172.21.15.27:6831/90668","heartbeat_front_addr":"172.21.15.27:6829/90668","state":["exists","up"]},{"osd":4,"uuid":"e1956a8e-e0f8-4e7c-acbb-40037415a354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":90863},{"type":"v1","addr":"172.21.15.124:6801","nonce":90863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":90863},{"type":"v1","addr":"172.21.15.124:6803","nonce":90863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":90863},{"type":"v1","addr":"172.21.15.124:6807","nonce":90863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":90863},{"type":"v1","addr":"172.21.15.124:6805","nonce":90863}]},"public_addr":"172.21.15.124:6801/90863","cluster_addr":"172.21.15.124:6803/90863","heartbeat_back_addr":"172.21.15.124:6807/90863","heartbeat_front_addr":"172.21.15.124:6805/90863","state":["exists","up"]},{"osd":5,"uuid":"306c59d4-846d-4b73-b1c9-051bec9e9368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":90885},{"type":"v1","addr":"172.21.15.124:6817","nonce":90885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":90885},{"type":"v1","addr":"172.21.15.124:6819","nonce":90885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":90885},{"type":"v1","addr":"172.21.15.124:6823","nonce":90885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":90885},{"type":"v1","addr":"172.21.15.124:6821","nonce":90885}]},"public_addr":"172.21.15.124:6817/90885","cluster_addr":"172.21.15.124:6819/90885","heartbeat_back_addr":"172.21.15.124:6823/90885","heartbeat_front_addr":"172.21.15.124:6821/90885","state":["exists","up"]},{"osd":6,"uuid":"15463e53-ac8d-47d1-b4f6-c0d31f29e4a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":90864},{"type":"v1","addr":"172.21.15.124:6809","nonce":90864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":90864},{"type":"v1","addr":"172.21.15.124:6811","nonce":90864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":90864},{"type":"v1","addr":"172.21.15.124:6815","nonce":90864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":90864},{"type":"v1","addr":"172.21.15.124:6813","nonce":90864}]},"public_addr":"172.21.15.124:6809/90864","cluster_addr":"172.21.15.124:6811/90864","heartbeat_back_addr":"172.21.15.124:6815/90864","heartbeat_front_addr":"172.21.15.124:6813/90864","state":["exists","up"]},{"osd":7,"uuid":"ae463a90-a9d3-498f-86fa-47a35b3fe889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":90894},{"type":"v1","addr":"172.21.15.124:6825","nonce":90894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":90894},{"type":"v1","addr":"172.21.15.124:6827","nonce":90894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":90894},{"type":"v1","addr":"172.21.15.124:6831","nonce":90894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":90894},{"type":"v1","addr":"172.21.15.124:6829","nonce":90894}]},"public_addr":"172.21.15.124:6825/90894","cluster_addr":"172.21.15.124:6827/90894","heartbeat_back_addr":"172.21.15.124:6831/90894","heartbeat_front_addr":"172.21.15.124:6829/90894","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T01:41:05.323 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.321+0000 7f006a925700 1 -- 172.21.15.27:0/926622459 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0050066ea0 msgr2=0x7f0050069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:05.323 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.321+0000 7f006a925700 1 --2- 172.21.15.27:0/926622459 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0050066ea0 0x7f0050069350 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f006415b1c0 tx=0x7f0054016000 comp rx=0 tx=0).stop 2023-10-06T01:41:05.323 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.321+0000 7f006a925700 1 -- 172.21.15.27:0/926622459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00640cd290 msgr2=0x7f00640c7820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:05.323 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.321+0000 7f006a925700 1 --2- 172.21.15.27:0/926622459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00640cd290 0x7f00640c7820 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f005c00bc60 tx=0x7f005c009840 comp rx=0 tx=0).stop 2023-10-06T01:41:05.323 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.321+0000 7f006a925700 1 -- 172.21.15.27:0/926622459 shutdown_connections 2023-10-06T01:41:05.324 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.321+0000 7f006a925700 1 --2- 172.21.15.27:0/926622459 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0050066ea0 0x7f0050069350 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:05.324 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.321+0000 7f006a925700 1 --2- 172.21.15.27:0/926622459 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f00640cd290 0x7f00640c7820 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:05.324 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.321+0000 7f006a925700 1 --2- 172.21.15.27:0/926622459 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f00640c9900 0x7f0064147ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:05.324 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.321+0000 7f006a925700 1 --2- 172.21.15.27:0/926622459 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f00640c9cd0 0x7f00640c5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:05.324 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.321+0000 7f006a925700 1 -- 172.21.15.27:0/926622459 >> 172.21.15.27:0/926622459 conn(0x7f0064068130 msgr2=0x7f0064156be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:05.324 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.321+0000 7f006a925700 1 -- 172.21.15.27:0/926622459 shutdown_connections 2023-10-06T01:41:05.324 INFO:teuthology.misc.health.smithi027.stderr:2023-10-06T01:41:05.321+0000 7f006a925700 1 -- 172.21.15.27:0/926622459 wait complete. 2023-10-06T01:41:05.333 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-10-06T01:41:05.333 INFO:tasks.ceph:Creating RBD pool 2023-10-06T01:41:05.334 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph osd pool create rbd 8 2023-10-06T01:41:05.452 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.451+0000 7f3767ef6700 1 Processor -- start 2023-10-06T01:41:05.452 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.451+0000 7f3767ef6700 1 -- start start 2023-10-06T01:41:05.454 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.452+0000 7f3767ef6700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f376006b470 0x7f376015d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:05.455 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.452+0000 7f3767ef6700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f376006f270 0x7f37600c79c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:05.455 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.452+0000 7f3767ef6700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f37600c7f00 0x7f37600aea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:05.455 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.452+0000 7f3767ef6700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f376015e290 con 0x7f376006f270 2023-10-06T01:41:05.455 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.452+0000 7f3767ef6700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37600c8870 con 0x7f37600c7f00 2023-10-06T01:41:05.455 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.452+0000 7f3767ef6700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37600ae340 con 0x7f376006b470 2023-10-06T01:41:05.455 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.452+0000 7f3765491700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f376006f270 0x7f37600c79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:05.455 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.452+0000 7f3765491700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f376006f270 0x7f37600c79c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40120/0 (socket says 172.21.15.27:40120) 2023-10-06T01:41:05.455 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.452+0000 7f3765491700 1 -- 172.21.15.27:0/1215796425 learned_addr learned my addr 172.21.15.27:0/1215796425 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:05.455 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.452+0000 7f3765c92700 1 --2- 172.21.15.27:0/1215796425 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f376006b470 0x7f376015d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:05.455 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.452+0000 7f3766493700 1 --2- 172.21.15.27:0/1215796425 >> v2:172.21.15.124:3300/0 conn(0x7f37600c7f00 0x7f37600aea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:05.455 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.452+0000 7f3765491700 1 -- 172.21.15.27:0/1215796425 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f376006b470 msgr2=0x7f376015d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:05.456 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.452+0000 7f3765491700 1 --2- 172.21.15.27:0/1215796425 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f376006b470 0x7f376015d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:05.456 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.452+0000 7f3765491700 1 -- 172.21.15.27:0/1215796425 >> v2:172.21.15.124:3300/0 conn(0x7f37600c7f00 msgr2=0x7f37600aea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:05.456 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.452+0000 7f3765491700 1 --2- 172.21.15.27:0/1215796425 >> v2:172.21.15.124:3300/0 conn(0x7f37600c7f00 0x7f37600aea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:05.456 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.452+0000 7f3765491700 1 -- 172.21.15.27:0/1215796425 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37600c8340 con 0x7f376006f270 2023-10-06T01:41:05.456 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.452+0000 7f3765491700 1 --2- 172.21.15.27:0/1215796425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f376006f270 0x7f37600c79c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f37600c7950 tx=0x7f3754009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb7343e0ab11961a server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:05.456 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.452+0000 7f3764c90700 1 -- 172.21.15.27:0/1215796425 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375400b640 con 0x7f376006f270 2023-10-06T01:41:05.456 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.452+0000 7f3764c90700 1 -- 172.21.15.27:0/1215796425 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f375400b7a0 con 0x7f376006f270 2023-10-06T01:41:05.456 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.452+0000 7f3764c90700 1 -- 172.21.15.27:0/1215796425 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375400bb60 con 0x7f376006f270 2023-10-06T01:41:05.456 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.453+0000 7f3767ef6700 1 -- 172.21.15.27:0/1215796425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f376006f270 msgr2=0x7f37600c79c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:05.456 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.453+0000 7f3767ef6700 1 --2- 172.21.15.27:0/1215796425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f376006f270 0x7f37600c79c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f37600c7950 tx=0x7f3754009250 comp rx=0 tx=0).stop 2023-10-06T01:41:05.456 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.453+0000 7f3767ef6700 1 -- 172.21.15.27:0/1215796425 shutdown_connections 2023-10-06T01:41:05.456 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.453+0000 7f3767ef6700 1 --2- 172.21.15.27:0/1215796425 >> v2:172.21.15.124:3300/0 conn(0x7f37600c7f00 0x7f37600aea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:05.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.453+0000 7f3767ef6700 1 --2- 172.21.15.27:0/1215796425 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f376006b470 0x7f376015d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:05.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.453+0000 7f3767ef6700 1 --2- 172.21.15.27:0/1215796425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f376006f270 0x7f37600c79c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:05.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.453+0000 7f3767ef6700 1 -- 172.21.15.27:0/1215796425 >> 172.21.15.27:0/1215796425 conn(0x7f376006e760 msgr2=0x7f376006eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:05.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.453+0000 7f3767ef6700 1 -- 172.21.15.27:0/1215796425 shutdown_connections 2023-10-06T01:41:05.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.453+0000 7f3767ef6700 1 -- 172.21.15.27:0/1215796425 wait complete. 2023-10-06T01:41:05.460 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.454+0000 7f3767ef6700 1 Processor -- start 2023-10-06T01:41:05.460 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.454+0000 7f3767ef6700 1 -- start start 2023-10-06T01:41:05.460 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.455+0000 7f3767ef6700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f376006b470 0x7f3760158280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:05.460 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.455+0000 7f3767ef6700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f376006f270 0x7f376006a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:05.460 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.455+0000 7f3767ef6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37600c7f00 0x7f376006a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:05.460 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.455+0000 7f3767ef6700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3760069280 con 0x7f37600c7f00 2023-10-06T01:41:05.460 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.455+0000 7f3767ef6700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37600c6330 con 0x7f376006b470 2023-10-06T01:41:05.460 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.455+0000 7f3767ef6700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37600c6610 con 0x7f376006f270 2023-10-06T01:41:05.460 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.455+0000 7f3766493700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37600c7f00 0x7f376006a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:05.460 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.455+0000 7f3766493700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37600c7f00 0x7f376006a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40130/0 (socket says 172.21.15.27:40130) 2023-10-06T01:41:05.461 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.455+0000 7f3766493700 1 -- 172.21.15.27:0/1312827688 learned_addr learned my addr 172.21.15.27:0/1312827688 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:05.461 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.455+0000 7f3765c92700 1 --2- 172.21.15.27:0/1312827688 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f376006b470 0x7f3760158280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:05.461 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.455+0000 7f3765491700 1 --2- 172.21.15.27:0/1312827688 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f376006f270 0x7f376006a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:05.461 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.455+0000 7f3766493700 1 -- 172.21.15.27:0/1312827688 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f376006f270 msgr2=0x7f376006a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:05.461 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.455+0000 7f3766493700 1 --2- 172.21.15.27:0/1312827688 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f376006f270 0x7f376006a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:05.461 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.455+0000 7f3766493700 1 -- 172.21.15.27:0/1312827688 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f376006b470 msgr2=0x7f3760158280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:05.461 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.455+0000 7f3766493700 1 --2- 172.21.15.27:0/1312827688 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f376006b470 0x7f3760158280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:05.461 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.455+0000 7f3766493700 1 -- 172.21.15.27:0/1312827688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37540094f0 con 0x7f37600c7f00 2023-10-06T01:41:05.461 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.455+0000 7f3765c92700 1 --2- 172.21.15.27:0/1312827688 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f376006b470 0x7f3760158280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:41:05.461 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.455+0000 7f3766493700 1 --2- 172.21.15.27:0/1312827688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37600c7f00 0x7f376006a600 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3750009df0 tx=0x7f375000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:05.461 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.456+0000 7f375dffb700 1 -- 172.21.15.27:0/1312827688 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375000cb50 con 0x7f37600c7f00 2023-10-06T01:41:05.461 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.456+0000 7f375dffb700 1 -- 172.21.15.27:0/1312827688 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f375000f040 con 0x7f37600c7f00 2023-10-06T01:41:05.462 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.456+0000 7f375dffb700 1 -- 172.21.15.27:0/1312827688 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3750014480 con 0x7f37600c7f00 2023-10-06T01:41:05.462 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.456+0000 7f3767ef6700 1 -- 172.21.15.27:0/1312827688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37600c68f0 con 0x7f37600c7f00 2023-10-06T01:41:05.462 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.456+0000 7f3767ef6700 1 -- 172.21.15.27:0/1312827688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37601587c0 con 0x7f37600c7f00 2023-10-06T01:41:05.462 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.457+0000 7f3767ef6700 1 -- 172.21.15.27:0/1312827688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37600b11a0 con 0x7f37600c7f00 2023-10-06T01:41:05.462 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.457+0000 7f375dffb700 1 -- 172.21.15.27:0/1312827688 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) v1 ==== 84684+0+0 (secure 0 0 0) 0x7f37500147f0 con 0x7f37600c7f00 2023-10-06T01:41:05.462 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.457+0000 7f375dffb700 1 --2- 172.21.15.27:0/1312827688 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f374c066ea0 0x7f374c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:05.462 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.458+0000 7f375dffb700 1 -- 172.21.15.27:0/1312827688 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4467+0+0 (secure 0 0 0) 0x7f3750084ac0 con 0x7f37600c7f00 2023-10-06T01:41:05.462 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.458+0000 7f3765c92700 1 --2- 172.21.15.27:0/1312827688 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f374c066ea0 0x7f374c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:05.462 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.458+0000 7f3765c92700 1 --2- 172.21.15.27:0/1312827688 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f374c066ea0 0x7f374c069350 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3758006f00 tx=0x7f3758008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:05.463 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.462+0000 7f375dffb700 1 -- 172.21.15.27:0/1312827688 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f37600b11a0 con 0x7f37600c7f00 2023-10-06T01:41:05.656 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:05.654+0000 7f3767ef6700 1 -- 172.21.15.27:0/1312827688 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f37600b11a0 con 0x7f37600c7f00 2023-10-06T01:41:06.254 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.253+0000 7f375dffb700 1 -- 172.21.15.27:0/1312827688 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v14) v1 ==== 109+0+0 (secure 0 0 0) 0x7f37600b11a0 con 0x7f37600c7f00 2023-10-06T01:41:06.257 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.255+0000 7f3767ef6700 1 -- 172.21.15.27:0/1312827688 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f374c066ea0 msgr2=0x7f374c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:06.257 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.255+0000 7f3767ef6700 1 --2- 172.21.15.27:0/1312827688 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f374c066ea0 0x7f374c069350 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3758006f00 tx=0x7f3758008040 comp rx=0 tx=0).stop 2023-10-06T01:41:06.257 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.255+0000 7f3767ef6700 1 -- 172.21.15.27:0/1312827688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37600c7f00 msgr2=0x7f376006a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:06.257 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.255+0000 7f3767ef6700 1 --2- 172.21.15.27:0/1312827688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37600c7f00 0x7f376006a600 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3750009df0 tx=0x7f375000b040 comp rx=0 tx=0).stop 2023-10-06T01:41:06.257 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.256+0000 7f3767ef6700 1 -- 172.21.15.27:0/1312827688 shutdown_connections 2023-10-06T01:41:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.256+0000 7f3767ef6700 1 --2- 172.21.15.27:0/1312827688 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f374c066ea0 0x7f374c069350 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.256+0000 7f3767ef6700 1 --2- 172.21.15.27:0/1312827688 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37600c7f00 0x7f376006a600 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.256+0000 7f3767ef6700 1 --2- 172.21.15.27:0/1312827688 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f376006b470 0x7f3760158280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.256+0000 7f3767ef6700 1 --2- 172.21.15.27:0/1312827688 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f376006f270 0x7f376006a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.256+0000 7f3767ef6700 1 -- 172.21.15.27:0/1312827688 >> 172.21.15.27:0/1312827688 conn(0x7f376006e760 msgr2=0x7f3760158b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.256+0000 7f3767ef6700 1 -- 172.21.15.27:0/1312827688 shutdown_connections 2023-10-06T01:41:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.256+0000 7f3767ef6700 1 -- 172.21.15.27:0/1312827688 wait complete. 2023-10-06T01:41:06.258 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'rbd' created 2023-10-06T01:41:06.269 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-10-06T01:41:06.390 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.388+0000 7fdde16ea700 1 Processor -- start 2023-10-06T01:41:06.390 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.388+0000 7fdde16ea700 1 -- start start 2023-10-06T01:41:06.394 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.389+0000 7fdde16ea700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fdddc157410 0x7fdddc0b16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:06.394 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.389+0000 7fdde16ea700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdddc0ccd60 0x7fdddc0b3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:06.394 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.389+0000 7fdde16ea700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdddc0b2fc0 con 0x7fdddc1577e0 2023-10-06T01:41:06.394 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.389+0000 7fdde16ea700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdddc0cd8e0 con 0x7fdddc157410 2023-10-06T01:41:06.394 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.389+0000 7fdde16ea700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc0cd320 con 0x7fdddc0ccd60 2023-10-06T01:41:06.394 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.389+0000 7fddda7fc700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdddc1577e0 0x7fdddc0b1c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47250/0 (socket says 172.21.15.27:47250) 2023-10-06T01:41:06.394 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.389+0000 7fddda7fc700 1 -- 172.21.15.27:0/2265348897 learned_addr learned my addr 172.21.15.27:0/2265348897 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:06.394 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.390+0000 7fdddb7fe700 1 --2- 172.21.15.27:0/2265348897 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdddc0ccd60 0x7fdddc0b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:06.394 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.390+0000 7fdddaffd700 1 --2- 172.21.15.27:0/2265348897 >> v2:172.21.15.124:3300/0 conn(0x7fdddc157410 0x7fdddc0b16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:06.394 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.390+0000 7fddd9ffb700 1 -- 172.21.15.27:0/2265348897 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3370094699 0 0) 0x7fdddc0b2fc0 con 0x7fdddc1577e0 2023-10-06T01:41:06.395 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.390+0000 7fddd9ffb700 1 -- 172.21.15.27:0/2265348897 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddc8003570 con 0x7fdddc1577e0 2023-10-06T01:41:06.395 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.390+0000 7fddd9ffb700 1 -- 172.21.15.27:0/2265348897 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 21925591 0 0) 0x7fddc8003570 con 0x7fdddc1577e0 2023-10-06T01:41:06.395 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.390+0000 7fddd9ffb700 1 -- 172.21.15.27:0/2265348897 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdddc0ccd60 msgr2=0x7fdddc0b3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:06.395 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.390+0000 7fddd9ffb700 1 --2- 172.21.15.27:0/2265348897 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdddc0ccd60 0x7fdddc0b3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:06.395 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.390+0000 7fddd9ffb700 1 -- 172.21.15.27:0/2265348897 >> v2:172.21.15.124:3300/0 conn(0x7fdddc157410 msgr2=0x7fdddc0b16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:06.395 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.390+0000 7fddd9ffb700 1 --2- 172.21.15.27:0/2265348897 >> v2:172.21.15.124:3300/0 conn(0x7fdddc157410 0x7fdddc0b16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:06.395 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.390+0000 7fddd9ffb700 1 -- 172.21.15.27:0/2265348897 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdddc0cd600 con 0x7fdddc1577e0 2023-10-06T01:41:06.395 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.390+0000 7fddd9ffb700 1 -- 172.21.15.27:0/2265348897 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2619078944 0 0) 0x7fddc4001470 con 0x7fdddc1577e0 2023-10-06T01:41:06.395 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.390+0000 7fddd9ffb700 1 -- 172.21.15.27:0/2265348897 >> v1:172.21.15.27:6789/0 conn(0x7fdddc1577e0 legacy=0x7fdddc0b1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:06.395 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.390+0000 7fddd9ffb700 1 --2- 172.21.15.27:0/2265348897 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fddc8003a60 0x7fddc8003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:06.395 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.390+0000 7fddd9ffb700 1 --2- 172.21.15.27:0/2265348897 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdddc0ccd60 0x7fddc8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:06.395 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.390+0000 7fddd9ffb700 1 --2- 172.21.15.27:0/2265348897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddc8004d50 0x7fddc80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:06.396 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.390+0000 7fddd9ffb700 1 -- 172.21.15.27:0/2265348897 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc0cd8e0 con 0x7fddc8004d50 2023-10-06T01:41:06.396 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.390+0000 7fddd9ffb700 1 -- 172.21.15.27:0/2265348897 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc0cd320 con 0x7fddc8003a60 2023-10-06T01:41:06.396 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.390+0000 7fddd9ffb700 1 -- 172.21.15.27:0/2265348897 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddc800bb10 con 0x7fdddc0ccd60 2023-10-06T01:41:06.396 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.390+0000 7fdddb7fe700 1 --2- 172.21.15.27:0/2265348897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddc8004d50 0x7fddc80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:06.396 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.390+0000 7fdddaffd700 1 --2- 172.21.15.27:0/2265348897 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fddc8003a60 0x7fddc8003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:06.396 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.390+0000 7fddda7fc700 1 --2- 172.21.15.27:0/2265348897 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdddc0ccd60 0x7fddc8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:06.396 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.390+0000 7fdddb7fe700 1 -- 172.21.15.27:0/2265348897 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdddc0ccd60 msgr2=0x7fddc8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:06.396 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.390+0000 7fdddb7fe700 1 --2- 172.21.15.27:0/2265348897 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdddc0ccd60 0x7fddc8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:06.396 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.390+0000 7fdddb7fe700 1 -- 172.21.15.27:0/2265348897 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fddc8003a60 msgr2=0x7fddc8003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:06.396 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.390+0000 7fdddb7fe700 1 --2- 172.21.15.27:0/2265348897 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fddc8003a60 0x7fddc8003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:06.396 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.390+0000 7fdddb7fe700 1 -- 172.21.15.27:0/2265348897 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc8003570 con 0x7fddc8004d50 2023-10-06T01:41:06.397 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.391+0000 7fdddb7fe700 1 --2- 172.21.15.27:0/2265348897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddc8004d50 0x7fddc80051c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fddcc00e210 tx=0x7fddcc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2ae9203db30b392 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:06.397 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.391+0000 7fddd9ffb700 1 -- 172.21.15.27:0/2265348897 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddcc00eac0 con 0x7fddc8004d50 2023-10-06T01:41:06.397 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.391+0000 7fddd9ffb700 1 -- 172.21.15.27:0/2265348897 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddcc00f040 con 0x7fddc8004d50 2023-10-06T01:41:06.397 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.391+0000 7fddd9ffb700 1 -- 172.21.15.27:0/2265348897 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddcc00ee70 con 0x7fddc8004d50 2023-10-06T01:41:06.397 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.391+0000 7fdde16ea700 1 -- 172.21.15.27:0/2265348897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddc8004d50 msgr2=0x7fddc80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:06.397 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.391+0000 7fdde16ea700 1 --2- 172.21.15.27:0/2265348897 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fddc8004d50 0x7fddc80051c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fddcc00e210 tx=0x7fddcc00b040 comp rx=0 tx=0).stop 2023-10-06T01:41:06.397 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.391+0000 7fdddb7fe700 1 -- 172.21.15.27:0/2265348897 reap_dead start 2023-10-06T01:41:06.397 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.391+0000 7fdde16ea700 1 -- 172.21.15.27:0/2265348897 shutdown_connections 2023-10-06T01:41:06.397 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.391+0000 7fdde16ea700 1 -- 172.21.15.27:0/2265348897 >> 172.21.15.27:0/2265348897 conn(0x7fdddc0ce4c0 msgr2=0x7fdddc0ce890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:06.397 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.391+0000 7fdde16ea700 1 -- 172.21.15.27:0/2265348897 shutdown_connections 2023-10-06T01:41:06.397 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.391+0000 7fdde16ea700 1 -- 172.21.15.27:0/2265348897 wait complete. 2023-10-06T01:41:06.397 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.392+0000 7fdde16ea700 1 Processor -- start 2023-10-06T01:41:06.397 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.393+0000 7fdde16ea700 1 -- start start 2023-10-06T01:41:06.400 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.393+0000 7fdde16ea700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fddc8004d50 0x7fdddc15ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:06.400 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.393+0000 7fdde16ea700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdddc168b40 0x7fdddc142fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:06.400 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.393+0000 7fdde16ea700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdddc15a7a0 0x7fdddc0b0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:06.400 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.393+0000 7fdde16ea700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc15ba80 con 0x7fdddc15a7a0 2023-10-06T01:41:06.400 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.393+0000 7fdde16ea700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc15c600 con 0x7fdddc168b40 2023-10-06T01:41:06.401 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.393+0000 7fdde16ea700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdddc15bcd0 con 0x7fddc8004d50 2023-10-06T01:41:06.401 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.393+0000 7fdddb7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdddc15a7a0 0x7fdddc0b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:06.401 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.393+0000 7fdddb7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdddc15a7a0 0x7fdddc0b0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40148/0 (socket says 172.21.15.27:40148) 2023-10-06T01:41:06.401 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.393+0000 7fdddb7fe700 1 -- 172.21.15.27:0/240868185 learned_addr learned my addr 172.21.15.27:0/240868185 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:06.401 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.393+0000 7fdddaffd700 1 --2- 172.21.15.27:0/240868185 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fddc8004d50 0x7fdddc15ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:06.401 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.393+0000 7fddda7fc700 1 --2- 172.21.15.27:0/240868185 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdddc168b40 0x7fdddc142fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:06.401 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.394+0000 7fdddb7fe700 1 -- 172.21.15.27:0/240868185 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fddc8004d50 msgr2=0x7fdddc15ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:06.401 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.394+0000 7fdddb7fe700 1 --2- 172.21.15.27:0/240868185 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fddc8004d50 0x7fdddc15ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:06.401 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.394+0000 7fdddb7fe700 1 -- 172.21.15.27:0/240868185 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdddc168b40 msgr2=0x7fdddc142fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:06.401 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.394+0000 7fdddb7fe700 1 --2- 172.21.15.27:0/240868185 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdddc168b40 0x7fdddc142fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:06.401 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.394+0000 7fdddb7fe700 1 -- 172.21.15.27:0/240868185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdddc15bfb0 con 0x7fdddc15a7a0 2023-10-06T01:41:06.401 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.394+0000 7fddda7fc700 1 --2- 172.21.15.27:0/240868185 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdddc168b40 0x7fdddc142fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:06.402 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.394+0000 7fdddb7fe700 1 --2- 172.21.15.27:0/240868185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdddc15a7a0 0x7fdddc0b0000 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fddcc002230 tx=0x7fddcc01a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:06.402 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.394+0000 7fddd2ffd700 1 -- 172.21.15.27:0/240868185 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddcc01d630 con 0x7fdddc15a7a0 2023-10-06T01:41:06.402 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.394+0000 7fddd2ffd700 1 -- 172.21.15.27:0/240868185 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddcc01d7b0 con 0x7fdddc15a7a0 2023-10-06T01:41:06.402 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.394+0000 7fdde16ea700 1 -- 172.21.15.27:0/240868185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdddc15c290 con 0x7fdddc15a7a0 2023-10-06T01:41:06.402 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.394+0000 7fddd2ffd700 1 -- 172.21.15.27:0/240868185 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddcc01db70 con 0x7fdddc15a7a0 2023-10-06T01:41:06.402 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.394+0000 7fdde16ea700 1 -- 172.21.15.27:0/240868185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdddc15ac20 con 0x7fdddc15a7a0 2023-10-06T01:41:06.402 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.395+0000 7fdde16ea700 1 -- 172.21.15.27:0/240868185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdddc15c7b0 con 0x7fdddc15a7a0 2023-10-06T01:41:06.402 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.395+0000 7fddd2ffd700 1 -- 172.21.15.27:0/240868185 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) v1 ==== 84684+0+0 (secure 0 0 0) 0x7fddcc003040 con 0x7fdddc15a7a0 2023-10-06T01:41:06.402 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.396+0000 7fddd2ffd700 1 --2- 172.21.15.27:0/240868185 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fddbc066ea0 0x7fddbc069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:06.402 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.396+0000 7fddd2ffd700 1 -- 172.21.15.27:0/240868185 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 5184+0+0 (secure 0 0 0) 0x7fddcc0857d0 con 0x7fdddc15a7a0 2023-10-06T01:41:06.402 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.396+0000 7fdddaffd700 1 --2- 172.21.15.27:0/240868185 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fddbc066ea0 0x7fddbc069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:06.402 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.397+0000 7fdddaffd700 1 --2- 172.21.15.27:0/240868185 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fddbc066ea0 0x7fddbc069350 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fddd4006f00 tx=0x7fddd40091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:06.403 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.401+0000 7fddd2ffd700 1 -- 172.21.15.27:0/240868185 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdddc15c7b0 con 0x7fdddc15a7a0 2023-10-06T01:41:06.594 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:06.593+0000 7fdde16ea700 1 -- 172.21.15.27:0/240868185 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fdddc0b2b90 con 0x7fdddc15a7a0 2023-10-06T01:41:07.254 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.253+0000 7fddd2ffd700 1 -- 172.21.15.27:0/240868185 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v15) v1 ==== 173+0+0 (secure 0 0 0) 0x7fddcc0180c0 con 0x7fdddc15a7a0 2023-10-06T01:41:07.257 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.256+0000 7fdde16ea700 1 -- 172.21.15.27:0/240868185 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fddbc066ea0 msgr2=0x7fddbc069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:07.257 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.256+0000 7fdde16ea700 1 --2- 172.21.15.27:0/240868185 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fddbc066ea0 0x7fddbc069350 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fddd4006f00 tx=0x7fddd40091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:07.257 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.256+0000 7fdde16ea700 1 -- 172.21.15.27:0/240868185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdddc15a7a0 msgr2=0x7fdddc0b0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:07.258 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.256+0000 7fdde16ea700 1 --2- 172.21.15.27:0/240868185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdddc15a7a0 0x7fdddc0b0000 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fddcc002230 tx=0x7fddcc01a000 comp rx=0 tx=0).stop 2023-10-06T01:41:07.258 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.256+0000 7fdde16ea700 1 -- 172.21.15.27:0/240868185 shutdown_connections 2023-10-06T01:41:07.258 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.256+0000 7fdde16ea700 1 --2- 172.21.15.27:0/240868185 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fddbc066ea0 0x7fddbc069350 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:07.258 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.256+0000 7fdde16ea700 1 --2- 172.21.15.27:0/240868185 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fddc8004d50 0x7fdddc15ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:07.258 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.256+0000 7fdde16ea700 1 --2- 172.21.15.27:0/240868185 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdddc15a7a0 0x7fdddc0b0000 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:07.258 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.256+0000 7fdde16ea700 1 --2- 172.21.15.27:0/240868185 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdddc168b40 0x7fdddc142fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:07.258 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.256+0000 7fdde16ea700 1 -- 172.21.15.27:0/240868185 >> 172.21.15.27:0/240868185 conn(0x7fdddc0ce4c0 msgr2=0x7fdddc153870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:07.258 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.256+0000 7fdde16ea700 1 -- 172.21.15.27:0/240868185 shutdown_connections 2023-10-06T01:41:07.258 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.256+0000 7fdde16ea700 1 -- 172.21.15.27:0/240868185 wait complete. 2023-10-06T01:41:07.259 INFO:teuthology.orchestra.run.smithi027.stderr:enabled application 'rbd' on pool 'rbd' 2023-10-06T01:41:07.272 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-06T01:41:07.272 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-06T01:41:07.272 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-06T01:41:07.274 INFO:tasks.ceph.mds.a:Started 2023-10-06T01:41:07.274 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-06T01:41:07.275 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-06T01:41:07.276 INFO:tasks.ceph.mds.c:Started 2023-10-06T01:41:07.277 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-06T01:41:07.277 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-06T01:41:07.279 INFO:tasks.ceph.mds.b:Started 2023-10-06T01:41:07.279 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-06T01:41:07.280 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-06T01:41:07.282 INFO:tasks.ceph.mds.d:Started 2023-10-06T01:41:07.282 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-10-06T01:41:07.282 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T01:41:07.358 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2023-10-06T01:41:07.359 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2023-10-06T01:41:07.364 INFO:tasks.ceph.mds.d.smithi124.stdout:starting mds.d at 2023-10-06T01:41:07.365 INFO:tasks.ceph.mds.b.smithi124.stdout:starting mds.b at 2023-10-06T01:41:07.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.415+0000 7f5cbeb2d700 1 Processor -- start 2023-10-06T01:41:07.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.415+0000 7f5cbeb2d700 1 -- start start 2023-10-06T01:41:07.418 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.416+0000 7f5cbeb2d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5cb815b170 0x7f5cb80cc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:07.418 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.416+0000 7f5cbeb2d700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cb80cad80 con 0x7f5cb80c8d60 2023-10-06T01:41:07.418 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.416+0000 7f5cbeb2d700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5cb80cd580 con 0x7f5cb80c9130 2023-10-06T01:41:07.418 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.416+0000 7f5cbeb2d700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb80cc970 con 0x7f5cb815b170 2023-10-06T01:41:07.418 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.416+0000 7f5cbc8c9700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5cb80c8d60 0x7f5cb80c3300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47278/0 (socket says 172.21.15.27:47278) 2023-10-06T01:41:07.419 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.416+0000 7f5cbc8c9700 1 -- 172.21.15.27:0/2108234016 learned_addr learned my addr 172.21.15.27:0/2108234016 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:07.419 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.416+0000 7f5cb7fff700 1 --2- 172.21.15.27:0/2108234016 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5cb815b170 0x7f5cb80cc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:07.419 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.416+0000 7f5cb77fe700 1 -- 172.21.15.27:0/2108234016 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 739192656 0 0) 0x7f5cb80cad80 con 0x7f5cb80c8d60 2023-10-06T01:41:07.419 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.416+0000 7f5cb77fe700 1 -- 172.21.15.27:0/2108234016 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ca8003570 con 0x7f5cb80c8d60 2023-10-06T01:41:07.419 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.416+0000 7f5cb77fe700 1 -- 172.21.15.27:0/2108234016 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2011611600 0 0) 0x7f5cb80cd580 con 0x7f5cb80c9130 2023-10-06T01:41:07.419 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.416+0000 7f5cb77fe700 1 -- 172.21.15.27:0/2108234016 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cb80cad80 con 0x7f5cb80c9130 2023-10-06T01:41:07.419 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.416+0000 7f5cb7fff700 1 -- 172.21.15.27:0/2108234016 >> v1:172.21.15.124:6789/0 conn(0x7f5cb80c9130 legacy=0x7f5cb8158270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:07.419 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.416+0000 7f5cb7fff700 1 -- 172.21.15.27:0/2108234016 >> v1:172.21.15.27:6789/0 conn(0x7f5cb80c8d60 legacy=0x7f5cb80c3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:07.419 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.416+0000 7f5cb77fe700 1 -- 172.21.15.27:0/2108234016 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2955887288 0 0) 0x7f5ca8003570 con 0x7f5cb80c8d60 2023-10-06T01:41:07.419 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.416+0000 7f5cb7fff700 1 -- 172.21.15.27:0/2108234016 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb80ccc50 con 0x7f5cb815b170 2023-10-06T01:41:07.419 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.417+0000 7f5cb7fff700 1 --2- 172.21.15.27:0/2108234016 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5cb815b170 0x7f5cb80cc2b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5ca4009910 tx=0x7f5ca40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62b3a58675a6f10 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:07.419 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.417+0000 7f5cb77fe700 1 -- 172.21.15.27:0/2108234016 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca4011630 con 0x7f5cb815b170 2023-10-06T01:41:07.420 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.417+0000 7f5cb77fe700 1 -- 172.21.15.27:0/2108234016 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ca4011790 con 0x7f5cb815b170 2023-10-06T01:41:07.420 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.417+0000 7f5cb77fe700 1 -- 172.21.15.27:0/2108234016 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca4011b30 con 0x7f5cb815b170 2023-10-06T01:41:07.422 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.417+0000 7f5cbeb2d700 1 -- 172.21.15.27:0/2108234016 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5cb815b170 msgr2=0x7f5cb80cc2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:07.422 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.417+0000 7f5cbeb2d700 1 --2- 172.21.15.27:0/2108234016 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5cb815b170 0x7f5cb80cc2b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5ca4009910 tx=0x7f5ca40091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:07.422 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.417+0000 7f5cbeb2d700 1 -- 172.21.15.27:0/2108234016 shutdown_connections 2023-10-06T01:41:07.423 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.417+0000 7f5cbeb2d700 1 --2- 172.21.15.27:0/2108234016 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5cb815b170 0x7f5cb80cc2b0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:07.423 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.417+0000 7f5cbeb2d700 1 -- 172.21.15.27:0/2108234016 >> 172.21.15.27:0/2108234016 conn(0x7f5cb80ce4d0 msgr2=0x7f5cb80ce8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:07.423 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.418+0000 7f5cbeb2d700 1 -- 172.21.15.27:0/2108234016 shutdown_connections 2023-10-06T01:41:07.423 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.418+0000 7f5cbeb2d700 1 -- 172.21.15.27:0/2108234016 wait complete. 2023-10-06T01:41:07.423 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.419+0000 7f5cbeb2d700 1 Processor -- start 2023-10-06T01:41:07.423 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.419+0000 7f5cbeb2d700 1 -- start start 2023-10-06T01:41:07.423 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.419+0000 7f5cbeb2d700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5cb80c8d60 0x7f5cb816bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:07.423 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.419+0000 7f5cbeb2d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cb80c9130 0x7f5cb80b32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:07.423 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.419+0000 7f5cbeb2d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5cb815b170 0x7f5cb8151460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:07.423 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.419+0000 7f5cbeb2d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb80b2e50 con 0x7f5cb80c9130 2023-10-06T01:41:07.423 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.420+0000 7f5cbeb2d700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb816c2c0 con 0x7f5cb80c8d60 2023-10-06T01:41:07.423 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.420+0000 7f5cbeb2d700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5cb80b2560 con 0x7f5cb815b170 2023-10-06T01:41:07.424 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.420+0000 7f5cb7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cb80c9130 0x7f5cb80b32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:07.424 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.420+0000 7f5cbc8c9700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5cb80c8d60 0x7f5cb816bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:07.424 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.420+0000 7f5cb7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cb80c9130 0x7f5cb80b32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40186/0 (socket says 172.21.15.27:40186) 2023-10-06T01:41:07.424 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.420+0000 7f5cb7fff700 1 -- 172.21.15.27:0/738163852 learned_addr learned my addr 172.21.15.27:0/738163852 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:07.424 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.420+0000 7f5cbd0ca700 1 --2- 172.21.15.27:0/738163852 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5cb815b170 0x7f5cb8151460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:07.424 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.420+0000 7f5cbc8c9700 1 -- 172.21.15.27:0/738163852 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5cb815b170 msgr2=0x7f5cb8151460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:07.424 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.420+0000 7f5cbc8c9700 1 --2- 172.21.15.27:0/738163852 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5cb815b170 0x7f5cb8151460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:07.424 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.420+0000 7f5cbc8c9700 1 -- 172.21.15.27:0/738163852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cb80c9130 msgr2=0x7f5cb80b32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:07.424 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.420+0000 7f5cbc8c9700 1 --2- 172.21.15.27:0/738163852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cb80c9130 0x7f5cb80b32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:07.424 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.420+0000 7f5cbc8c9700 1 -- 172.21.15.27:0/738163852 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca4009420 con 0x7f5cb80c8d60 2023-10-06T01:41:07.424 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.420+0000 7f5cb7fff700 1 --2- 172.21.15.27:0/738163852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cb80c9130 0x7f5cb80b32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:41:07.424 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.420+0000 7f5cbc8c9700 1 --2- 172.21.15.27:0/738163852 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5cb80c8d60 0x7f5cb816bd80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5cac007fd0 tx=0x7f5cac007ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:07.425 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.420+0000 7f5cb4ff9700 1 -- 172.21.15.27:0/738163852 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cac007360 con 0x7f5cb80c8d60 2023-10-06T01:41:07.425 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.420+0000 7f5cb4ff9700 1 -- 172.21.15.27:0/738163852 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cac0074e0 con 0x7f5cb80c8d60 2023-10-06T01:41:07.425 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.420+0000 7f5cbeb2d700 1 -- 172.21.15.27:0/738163852 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cb80b2800 con 0x7f5cb80c8d60 2023-10-06T01:41:07.425 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.420+0000 7f5cb4ff9700 1 -- 172.21.15.27:0/738163852 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cac004830 con 0x7f5cb80c8d60 2023-10-06T01:41:07.425 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.420+0000 7f5cbeb2d700 1 -- 172.21.15.27:0/738163852 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cb80c5b40 con 0x7f5cb80c8d60 2023-10-06T01:41:07.426 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.421+0000 7f5cbeb2d700 1 -- 172.21.15.27:0/738163852 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5cb80b3130 con 0x7f5cb80c8d60 2023-10-06T01:41:07.426 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.422+0000 7f5cb4ff9700 1 -- 172.21.15.27:0/738163852 <== mon.1 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 4) v1 ==== 84684+0+0 (secure 0 0 0) 0x7f5cac004ba0 con 0x7f5cb80c8d60 2023-10-06T01:41:07.426 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.422+0000 7f5cb4ff9700 1 --2- 172.21.15.27:0/738163852 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5c9c066ea0 0x7f5c9c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:07.426 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.422+0000 7f5cb4ff9700 1 -- 172.21.15.27:0/738163852 <== mon.1 v2:172.21.15.124:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f5cac083ed0 con 0x7f5cb80c8d60 2023-10-06T01:41:07.426 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.422+0000 7f5cb7fff700 1 --2- 172.21.15.27:0/738163852 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5c9c066ea0 0x7f5c9c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:07.426 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.423+0000 7f5cb7fff700 1 --2- 172.21.15.27:0/738163852 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5c9c066ea0 0x7f5c9c069350 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5ca4009fd0 tx=0x7f5ca4017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:07.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.427+0000 7f5cb4ff9700 1 -- 172.21.15.27:0/738163852 <== mon.1 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5cac051490 con 0x7f5cb80c8d60 2023-10-06T01:41:07.629 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.628+0000 7f5cbeb2d700 1 -- 172.21.15.27:0/738163852 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5cb8064050 con 0x7f5cb80c8d60 2023-10-06T01:41:07.631 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.629+0000 7f5cb4ff9700 1 -- 172.21.15.27:0/738163852 <== mon.1 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+13674 (secure 0 0 0) 0x7f5cac007c00 con 0x7f5cb80c8d60 2023-10-06T01:41:07.631 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:07.631 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":15,"fsid":"20a7bff0-74f8-4f2f-b6c6-899e804bc965","created":"2023-10-06T01:40:54.474215+0000","modified":"2023-10-06T01:41:07.251081+0000","last_up_change":"2023-10-06T01:41:05.242847+0000","last_in_change":"2023-10-06T01:40:56.890960+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-10-06T01:41:05.685386+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-10-06T01:41:06.092995+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b902c86d-800c-40c5-b152-b545e50d02c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":90644},{"type":"v1","addr":"172.21.15.27:6801","nonce":90644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":90644},{"type":"v1","addr":"172.21.15.27:6803","nonce":90644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":90644},{"type":"v1","addr":"172.21.15.27:6807","nonce":90644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":90644},{"type":"v1","addr":"172.21.15.27:6805","nonce":90644}]},"public_addr":"172.21.15.27:6801/90644","cluster_addr":"172.21.15.27:6803/90644","heartbeat_back_addr":"172.21.15.27:6807/90644","heartbeat_front_addr":"172.21.15.27:6805/90644","state":["exists","up"]},{"osd":1,"uuid":"5b4a70e5-b06e-48cd-bfc4-7962c5d05386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":90643},{"type":"v1","addr":"172.21.15.27:6809","nonce":90643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":90643},{"type":"v1","addr":"172.21.15.27:6811","nonce":90643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":90643},{"type":"v1","addr":"172.21.15.27:6815","nonce":90643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":90643},{"type":"v1","addr":"172.21.15.27:6813","nonce":90643}]},"public_addr":"172.21.15.27:6809/90643","cluster_addr":"172.21.15.27:6811/90643","heartbeat_back_addr":"172.21.15.27:6815/90643","heartbeat_front_addr":"172.21.15.27:6813/90643","state":["exists","up"]},{"osd":2,"uuid":"22d9de46-4073-4336-9122-5227e601eca0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":90645},{"type":"v1","addr":"172.21.15.27:6817","nonce":90645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":90645},{"type":"v1","addr":"172.21.15.27:6819","nonce":90645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":90645},{"type":"v1","addr":"172.21.15.27:6823","nonce":90645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":90645},{"type":"v1","addr":"172.21.15.27:6821","nonce":90645}]},"public_addr":"172.21.15.27:6817/90645","cluster_addr":"172.21.15.27:6819/90645","heartbeat_back_addr":"172.21.15.27:6823/90645","heartbeat_front_addr":"172.21.15.27:6821/90645","state":["exists","up"]},{"osd":3,"uuid":"438bb2d3-34e2-4b7c-9734-8360c1b67650","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":90668},{"type":"v1","addr":"172.21.15.27:6825","nonce":90668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":90668},{"type":"v1","addr":"172.21.15.27:6827","nonce":90668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":90668},{"type":"v1","addr":"172.21.15.27:6831","nonce":90668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":90668},{"type":"v1","addr":"172.21.15.27:6829","nonce":90668}]},"public_addr":"172.21.15.27:6825/90668","cluster_addr":"172.21.15.27:6827/90668","heartbeat_back_addr":"172.21.15.27:6831/90668","heartbeat_front_addr":"172.21.15.27:6829/90668","state":["exists","up"]},{"osd":4,"uuid":"e1956a8e-e0f8-4e7c-acbb-40037415a354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":90863},{"type":"v1","addr":"172.21.15.124:6801","nonce":90863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":90863},{"type":"v1","addr":"172.21.15.124:6803","nonce":90863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":90863},{"type":"v1","addr":"172.21.15.124:6807","nonce":90863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":90863},{"type":"v1","addr":"172.21.15.124:6805","nonce":90863}]},"public_addr":"172.21.15.124:6801/90863","cluster_addr":"172.21.15.124:6803/90863","heartbeat_back_addr":"172.21.15.124:6807/90863","heartbeat_front_addr":"172.21.15.124:6805/90863","state":["exists","up"]},{"osd":5,"uuid":"306c59d4-846d-4b73-b1c9-051bec9e9368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":90885},{"type":"v1","addr":"172.21.15.124:6817","nonce":90885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":90885},{"type":"v1","addr":"172.21.15.124:6819","nonce":90885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":90885},{"type":"v1","addr":"172.21.15.124:6823","nonce":90885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":90885},{"type":"v1","addr":"172.21.15.124:6821","nonce":90885}]},"public_addr":"172.21.15.124:6817/90885","cluster_addr":"172.21.15.124:6819/90885","heartbeat_back_addr":"172.21.15.124:6823/90885","heartbeat_front_addr":"172.21.15.124:6821/90885","state":["exists","up"]},{"osd":6,"uuid":"15463e53-ac8d-47d1-b4f6-c0d31f29e4a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":90864},{"type":"v1","addr":"172.21.15.124:6809","nonce":90864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":90864},{"type":"v1","addr":"172.21.15.124:6811","nonce":90864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":90864},{"type":"v1","addr":"172.21.15.124:6815","nonce":90864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":90864},{"type":"v1","addr":"172.21.15.124:6813","nonce":90864}]},"public_addr":"172.21.15.124:6809/90864","cluster_addr":"172.21.15.124:6811/90864","heartbeat_back_addr":"172.21.15.124:6815/90864","heartbeat_front_addr":"172.21.15.124:6813/90864","state":["exists","up"]},{"osd":7,"uuid":"ae463a90-a9d3-498f-86fa-47a35b3fe889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":90894},{"type":"v1","addr":"172.21.15.124:6825","nonce":90894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":90894},{"type":"v1","addr":"172.21.15.124:6827","nonce":90894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":90894},{"type":"v1","addr":"172.21.15.124:6831","nonce":90894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":90894},{"type":"v1","addr":"172.21.15.124:6829","nonce":90894}]},"public_addr":"172.21.15.124:6825/90894","cluster_addr":"172.21.15.124:6827/90894","heartbeat_back_addr":"172.21.15.124:6831/90894","heartbeat_front_addr":"172.21.15.124:6829/90894","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.827704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.839951+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.032230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.050605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.987908+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.031290+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.080707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.053692+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T01:41:07.633 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.632+0000 7f5cbeb2d700 1 -- 172.21.15.27:0/738163852 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5c9c066ea0 msgr2=0x7f5c9c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:07.633 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.632+0000 7f5cbeb2d700 1 --2- 172.21.15.27:0/738163852 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5c9c066ea0 0x7f5c9c069350 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5ca4009fd0 tx=0x7f5ca4017000 comp rx=0 tx=0).stop 2023-10-06T01:41:07.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.632+0000 7f5cbeb2d700 1 -- 172.21.15.27:0/738163852 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5cb80c8d60 msgr2=0x7f5cb816bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:07.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.632+0000 7f5cbeb2d700 1 --2- 172.21.15.27:0/738163852 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5cb80c8d60 0x7f5cb816bd80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5cac007fd0 tx=0x7f5cac007ac0 comp rx=0 tx=0).stop 2023-10-06T01:41:07.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.632+0000 7f5cbeb2d700 1 -- 172.21.15.27:0/738163852 shutdown_connections 2023-10-06T01:41:07.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.632+0000 7f5cbeb2d700 1 --2- 172.21.15.27:0/738163852 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5c9c066ea0 0x7f5c9c069350 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:07.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.632+0000 7f5cbeb2d700 1 --2- 172.21.15.27:0/738163852 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5cb815b170 0x7f5cb8151460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:07.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.632+0000 7f5cbeb2d700 1 --2- 172.21.15.27:0/738163852 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5cb80c8d60 0x7f5cb816bd80 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:07.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.632+0000 7f5cbeb2d700 1 --2- 172.21.15.27:0/738163852 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5cb80c9130 0x7f5cb80b32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:07.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.632+0000 7f5cbeb2d700 1 -- 172.21.15.27:0/738163852 >> 172.21.15.27:0/738163852 conn(0x7f5cb80ce4d0 msgr2=0x7f5cb8158410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:07.635 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.632+0000 7f5cbeb2d700 1 -- 172.21.15.27:0/738163852 shutdown_connections 2023-10-06T01:41:07.635 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.632+0000 7f5cbeb2d700 1 -- 172.21.15.27:0/738163852 wait complete. 2023-10-06T01:41:07.645 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-10-06T01:41:05.685386+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-10-06T01:41:06.092995+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-10-06T01:41:07.646 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-06T01:41:07.770 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.769+0000 7f7de7d87700 1 Processor -- start 2023-10-06T01:41:07.770 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.769+0000 7f7de7d87700 1 -- start start 2023-10-06T01:41:07.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.770+0000 7f7de7d87700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7de0159c50 0x7f7de015e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:07.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.770+0000 7f7de7d87700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7de00694f0 0x7f7de00aa970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:07.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.770+0000 7f7de7d87700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7de00ab0c0 con 0x7f7de0069120 2023-10-06T01:41:07.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.770+0000 7f7de7d87700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7de00aaeb0 con 0x7f7de0159c50 2023-10-06T01:41:07.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.770+0000 7f7de7d87700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7de00ab3a0 con 0x7f7de00694f0 2023-10-06T01:41:07.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.770+0000 7f7de5322700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7de0159c50 0x7f7de015e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:07.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.770+0000 7f7de5322700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7de0159c50 0x7f7de015e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40202/0 (socket says 172.21.15.27:40202) 2023-10-06T01:41:07.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.770+0000 7f7de5322700 1 -- 172.21.15.27:0/482162432 learned_addr learned my addr 172.21.15.27:0/482162432 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:07.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.770+0000 7f7de6324700 1 --2- 172.21.15.27:0/482162432 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7de00694f0 0x7f7de00aa970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:07.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.770+0000 7f7de5322700 1 -- 172.21.15.27:0/482162432 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7de00694f0 msgr2=0x7f7de00aa970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:07.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.770+0000 7f7de5322700 1 --2- 172.21.15.27:0/482162432 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7de00694f0 0x7f7de00aa970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:07.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.770+0000 7f7de5322700 1 -- 172.21.15.27:0/482162432 >> v1:172.21.15.124:6789/0 conn(0x7f7de0069120 legacy=0x7f7de0159540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:07.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.770+0000 7f7de5322700 1 -- 172.21.15.27:0/482162432 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de00ab680 con 0x7f7de0159c50 2023-10-06T01:41:07.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.770+0000 7f7de5322700 1 --2- 172.21.15.27:0/482162432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7de0159c50 0x7f7de015e180 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7dd00099e0 tx=0x7f7dd0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f706216b8ae37b18 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:07.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.771+0000 7f7de4b21700 1 -- 172.21.15.27:0/482162432 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd000b820 con 0x7f7de0159c50 2023-10-06T01:41:07.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.771+0000 7f7de4b21700 1 -- 172.21.15.27:0/482162432 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dd000b980 con 0x7f7de0159c50 2023-10-06T01:41:07.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.771+0000 7f7de4b21700 1 -- 172.21.15.27:0/482162432 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd000bd20 con 0x7f7de0159c50 2023-10-06T01:41:07.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.771+0000 7f7de7d87700 1 -- 172.21.15.27:0/482162432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7de0159c50 msgr2=0x7f7de015e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:07.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.771+0000 7f7de7d87700 1 --2- 172.21.15.27:0/482162432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7de0159c50 0x7f7de015e180 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7dd00099e0 tx=0x7f7dd0009250 comp rx=0 tx=0).stop 2023-10-06T01:41:07.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.771+0000 7f7de7d87700 1 -- 172.21.15.27:0/482162432 shutdown_connections 2023-10-06T01:41:07.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.771+0000 7f7de7d87700 1 --2- 172.21.15.27:0/482162432 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7de00694f0 0x7f7de00aa970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:07.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.771+0000 7f7de7d87700 1 --2- 172.21.15.27:0/482162432 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7de0159c50 0x7f7de015e180 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:07.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.771+0000 7f7de7d87700 1 -- 172.21.15.27:0/482162432 >> 172.21.15.27:0/482162432 conn(0x7f7de006e910 msgr2=0x7f7de006be30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:07.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.771+0000 7f7de7d87700 1 -- 172.21.15.27:0/482162432 shutdown_connections 2023-10-06T01:41:07.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.771+0000 7f7de7d87700 1 -- 172.21.15.27:0/482162432 wait complete. 2023-10-06T01:41:07.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.772+0000 7f7de7d87700 1 Processor -- start 2023-10-06T01:41:07.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.772+0000 7f7de7d87700 1 -- start start 2023-10-06T01:41:07.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.773+0000 7f7de7d87700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7de0069120 0x7f7de014f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:07.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.773+0000 7f7de7d87700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7de00694f0 0x7f7de0157f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:07.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.773+0000 7f7de7d87700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7de0159c50 0x7f7de01584d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:07.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.773+0000 7f7de7d87700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de006aad0 con 0x7f7de0159c50 2023-10-06T01:41:07.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.773+0000 7f7de7d87700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de0155ad0 con 0x7f7de0069120 2023-10-06T01:41:07.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.773+0000 7f7de7d87700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7de006a1a0 con 0x7f7de00694f0 2023-10-06T01:41:07.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.773+0000 7f7de6324700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7de0159c50 0x7f7de01584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:07.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.773+0000 7f7de5b23700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7de0069120 0x7f7de014f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:07.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.773+0000 7f7de6324700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7de0159c50 0x7f7de01584d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40210/0 (socket says 172.21.15.27:40210) 2023-10-06T01:41:07.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.773+0000 7f7de6324700 1 -- 172.21.15.27:0/2728356837 learned_addr learned my addr 172.21.15.27:0/2728356837 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:07.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.773+0000 7f7de5322700 1 --2- 172.21.15.27:0/2728356837 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7de00694f0 0x7f7de0157f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:07.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.773+0000 7f7de6324700 1 -- 172.21.15.27:0/2728356837 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7de00694f0 msgr2=0x7f7de0157f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:07.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.773+0000 7f7de6324700 1 --2- 172.21.15.27:0/2728356837 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7de00694f0 0x7f7de0157f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:07.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.773+0000 7f7de6324700 1 -- 172.21.15.27:0/2728356837 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7de0069120 msgr2=0x7f7de014f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:07.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.773+0000 7f7de6324700 1 --2- 172.21.15.27:0/2728356837 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7de0069120 0x7f7de014f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:07.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.773+0000 7f7de6324700 1 -- 172.21.15.27:0/2728356837 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd00094f0 con 0x7f7de0159c50 2023-10-06T01:41:07.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.774+0000 7f7de6324700 1 --2- 172.21.15.27:0/2728356837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7de0159c50 0x7f7de01584d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7dd8007f20 tx=0x7f7dd8009860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:07.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.774+0000 7f7dd5ffb700 1 -- 172.21.15.27:0/2728356837 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd800eb90 con 0x7f7de0159c50 2023-10-06T01:41:07.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.774+0000 7f7dd5ffb700 1 -- 172.21.15.27:0/2728356837 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dd800ed10 con 0x7f7de0159c50 2023-10-06T01:41:07.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.774+0000 7f7de7d87700 1 -- 172.21.15.27:0/2728356837 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7de006a480 con 0x7f7de0159c50 2023-10-06T01:41:07.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.774+0000 7f7dd5ffb700 1 -- 172.21.15.27:0/2728356837 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd8014670 con 0x7f7de0159c50 2023-10-06T01:41:07.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.774+0000 7f7de7d87700 1 -- 172.21.15.27:0/2728356837 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7de014f8c0 con 0x7f7de0159c50 2023-10-06T01:41:07.779 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.775+0000 7f7dd5ffb700 1 -- 172.21.15.27:0/2728356837 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) v1 ==== 84684+0+0 (secure 0 0 0) 0x7f7dd80149e0 con 0x7f7de0159c50 2023-10-06T01:41:07.779 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.775+0000 7f7dd5ffb700 1 --2- 172.21.15.27:0/2728356837 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7dc4066ea0 0x7f7dc4069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:07.779 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.776+0000 7f7dd5ffb700 1 -- 172.21.15.27:0/2728356837 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f7dd80847a0 con 0x7f7de0159c50 2023-10-06T01:41:07.780 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.776+0000 7f7de5b23700 1 --2- 172.21.15.27:0/2728356837 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7dc4066ea0 0x7f7dc4069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:07.780 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.776+0000 7f7de7d87700 1 -- 172.21.15.27:0/2728356837 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7de006ac40 con 0x7f7de0159c50 2023-10-06T01:41:07.780 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.776+0000 7f7de5b23700 1 --2- 172.21.15.27:0/2728356837 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7dc4066ea0 0x7f7dc4069350 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7de014ffe0 tx=0x7f7ddc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:07.781 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.780+0000 7f7dd5ffb700 1 -- 172.21.15.27:0/2728356837 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7dd8051ce0 con 0x7f7de0159c50 2023-10-06T01:41:07.979 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.977+0000 7f7de7d87700 1 -- 172.21.15.27:0/2728356837 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f7de014fab0 con 0x7f7de0159c50 2023-10-06T01:41:07.979 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.978+0000 7f7dd5ffb700 1 -- 172.21.15.27:0/2728356837 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v15) v1 ==== 92+0+10 (secure 0 0 0) 0x7f7dd8055300 con 0x7f7de0159c50 2023-10-06T01:41:07.979 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 8 2023-10-06T01:41:07.981 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.979+0000 7f7de7d87700 1 -- 172.21.15.27:0/2728356837 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7dc4066ea0 msgr2=0x7f7dc4069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:07.981 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.979+0000 7f7de7d87700 1 --2- 172.21.15.27:0/2728356837 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7dc4066ea0 0x7f7dc4069350 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7de014ffe0 tx=0x7f7ddc009040 comp rx=0 tx=0).stop 2023-10-06T01:41:07.981 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.979+0000 7f7de7d87700 1 -- 172.21.15.27:0/2728356837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7de0159c50 msgr2=0x7f7de01584d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:07.981 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.979+0000 7f7de7d87700 1 --2- 172.21.15.27:0/2728356837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7de0159c50 0x7f7de01584d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7dd8007f20 tx=0x7f7dd8009860 comp rx=0 tx=0).stop 2023-10-06T01:41:07.981 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.979+0000 7f7de7d87700 1 -- 172.21.15.27:0/2728356837 shutdown_connections 2023-10-06T01:41:07.982 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.979+0000 7f7de7d87700 1 --2- 172.21.15.27:0/2728356837 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7dc4066ea0 0x7f7dc4069350 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:07.982 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.979+0000 7f7de7d87700 1 --2- 172.21.15.27:0/2728356837 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7de0159c50 0x7f7de01584d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:07.982 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.979+0000 7f7de7d87700 1 --2- 172.21.15.27:0/2728356837 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7de0069120 0x7f7de014f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:07.982 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.979+0000 7f7de7d87700 1 --2- 172.21.15.27:0/2728356837 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7de00694f0 0x7f7de0157f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:07.982 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.980+0000 7f7de7d87700 1 -- 172.21.15.27:0/2728356837 >> 172.21.15.27:0/2728356837 conn(0x7f7de006e910 msgr2=0x7f7de00af8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:07.982 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.980+0000 7f7de7d87700 1 -- 172.21.15.27:0/2728356837 shutdown_connections 2023-10-06T01:41:07.982 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:07.980+0000 7f7de7d87700 1 -- 172.21.15.27:0/2728356837 wait complete. 2023-10-06T01:41:07.993 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-06T01:41:08.114 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.113+0000 7f2bd2a2d700 1 Processor -- start 2023-10-06T01:41:08.115 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.113+0000 7f2bd2a2d700 1 -- start start 2023-10-06T01:41:08.117 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.114+0000 7f2bd2a2d700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f2bcc142e90 0x7f2bcc069810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:08.117 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.114+0000 7f2bd2a2d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2bcc15d9a0 0x7f2bcc0b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:08.117 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.114+0000 7f2bd2a2d700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bcc069380 con 0x7f2bcc069d50 2023-10-06T01:41:08.117 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.114+0000 7f2bd2a2d700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bcc068770 con 0x7f2bcc15d9a0 2023-10-06T01:41:08.117 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.114+0000 7f2bd2a2d700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bcc068a50 con 0x7f2bcc142e90 2023-10-06T01:41:08.117 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.114+0000 7f2bd222c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2bcc15d9a0 0x7f2bcc0b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:08.117 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.114+0000 7f2bd122a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2bcc069d50 0x7f2bcc15d290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47290/0 (socket says 172.21.15.27:47290) 2023-10-06T01:41:08.117 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.114+0000 7f2bd222c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2bcc15d9a0 0x7f2bcc0b5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40214/0 (socket says 172.21.15.27:40214) 2023-10-06T01:41:08.118 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.114+0000 7f2bd122a700 1 -- 172.21.15.27:0/3439893336 learned_addr learned my addr 172.21.15.27:0/3439893336 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:08.118 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.114+0000 7f2bd1a2b700 1 --2- 172.21.15.27:0/3439893336 >> v2:172.21.15.124:3300/0 conn(0x7f2bcc142e90 0x7f2bcc069810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:08.118 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.114+0000 7f2bd222c700 1 -- 172.21.15.27:0/3439893336 >> v2:172.21.15.124:3300/0 conn(0x7f2bcc142e90 msgr2=0x7f2bcc069810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:08.118 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.114+0000 7f2bd222c700 1 --2- 172.21.15.27:0/3439893336 >> v2:172.21.15.124:3300/0 conn(0x7f2bcc142e90 0x7f2bcc069810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:08.118 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.114+0000 7f2bd222c700 1 -- 172.21.15.27:0/3439893336 >> v1:172.21.15.27:6789/0 conn(0x7f2bcc069d50 legacy=0x7f2bcc15d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:08.118 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.114+0000 7f2bd0a29700 1 -- 172.21.15.27:0/3439893336 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 930809810 0 0) 0x7f2bcc069380 con 0x7f2bcc069d50 2023-10-06T01:41:08.118 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.114+0000 7f2bd222c700 1 -- 172.21.15.27:0/3439893336 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bcc068d30 con 0x7f2bcc15d9a0 2023-10-06T01:41:08.118 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.115+0000 7f2bd222c700 1 --2- 172.21.15.27:0/3439893336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bcc15d9a0 0x7f2bcc0b5c90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc089390 tx=0x7f2bbc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79dccd8f3708b31 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:08.118 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.115+0000 7f2bd0a29700 1 -- 172.21.15.27:0/3439893336 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc00e070 con 0x7f2bcc15d9a0 2023-10-06T01:41:08.118 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.115+0000 7f2bd0a29700 1 -- 172.21.15.27:0/3439893336 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bbc00bb60 con 0x7f2bcc15d9a0 2023-10-06T01:41:08.118 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.115+0000 7f2bd0a29700 1 -- 172.21.15.27:0/3439893336 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bbc012430 con 0x7f2bcc15d9a0 2023-10-06T01:41:08.119 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.115+0000 7f2bd2a2d700 1 -- 172.21.15.27:0/3439893336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bcc15d9a0 msgr2=0x7f2bcc0b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:08.119 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.115+0000 7f2bd2a2d700 1 --2- 172.21.15.27:0/3439893336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bcc15d9a0 0x7f2bcc0b5c90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc089390 tx=0x7f2bbc009780 comp rx=0 tx=0).stop 2023-10-06T01:41:08.119 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.115+0000 7f2bd2a2d700 1 -- 172.21.15.27:0/3439893336 shutdown_connections 2023-10-06T01:41:08.119 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.115+0000 7f2bd2a2d700 1 --2- 172.21.15.27:0/3439893336 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bcc15d9a0 0x7f2bcc0b5c90 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:08.119 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.115+0000 7f2bd2a2d700 1 --2- 172.21.15.27:0/3439893336 >> v2:172.21.15.124:3300/0 conn(0x7f2bcc142e90 0x7f2bcc069810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:08.119 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.115+0000 7f2bd2a2d700 1 -- 172.21.15.27:0/3439893336 >> 172.21.15.27:0/3439893336 conn(0x7f2bcc0ccfe0 msgr2=0x7f2bcc0cd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:08.119 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.115+0000 7f2bd2a2d700 1 -- 172.21.15.27:0/3439893336 shutdown_connections 2023-10-06T01:41:08.119 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.115+0000 7f2bd2a2d700 1 -- 172.21.15.27:0/3439893336 wait complete. 2023-10-06T01:41:08.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.116+0000 7f2bd2a2d700 1 Processor -- start 2023-10-06T01:41:08.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.117+0000 7f2bd2a2d700 1 -- start start 2023-10-06T01:41:08.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.117+0000 7f2bd2a2d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2bcc069d50 0x7f2bcc13f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:08.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.117+0000 7f2bd2a2d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bcc142e90 0x7f2bcc15b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:08.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.117+0000 7f2bd2a2d700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2bcc15d9a0 0x7f2bcc16aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:08.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.117+0000 7f2bd2a2d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bcc15b990 con 0x7f2bcc142e90 2023-10-06T01:41:08.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.117+0000 7f2bd2a2d700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bcc15a730 con 0x7f2bcc15d9a0 2023-10-06T01:41:08.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.117+0000 7f2bd2a2d700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bcc15afc0 con 0x7f2bcc069d50 2023-10-06T01:41:08.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.117+0000 7f2bd122a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bcc142e90 0x7f2bcc15b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:08.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.117+0000 7f2bd122a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bcc142e90 0x7f2bcc15b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40226/0 (socket says 172.21.15.27:40226) 2023-10-06T01:41:08.123 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.117+0000 7f2bd122a700 1 -- 172.21.15.27:0/2524561242 learned_addr learned my addr 172.21.15.27:0/2524561242 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:08.123 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.117+0000 7f2bd1a2b700 1 --2- 172.21.15.27:0/2524561242 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2bcc069d50 0x7f2bcc13f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:08.123 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.117+0000 7f2bd222c700 1 --2- 172.21.15.27:0/2524561242 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2bcc15d9a0 0x7f2bcc16aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:08.123 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.118+0000 7f2bd122a700 1 -- 172.21.15.27:0/2524561242 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2bcc069d50 msgr2=0x7f2bcc13f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:08.123 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.118+0000 7f2bd122a700 1 --2- 172.21.15.27:0/2524561242 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2bcc069d50 0x7f2bcc13f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:08.123 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.118+0000 7f2bd122a700 1 -- 172.21.15.27:0/2524561242 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2bcc15d9a0 msgr2=0x7f2bcc16aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:08.123 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.118+0000 7f2bd122a700 1 --2- 172.21.15.27:0/2524561242 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2bcc15d9a0 0x7f2bcc16aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:08.123 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.118+0000 7f2bd122a700 1 -- 172.21.15.27:0/2524561242 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bbc009a20 con 0x7f2bcc142e90 2023-10-06T01:41:08.123 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.118+0000 7f2bd122a700 1 --2- 172.21.15.27:0/2524561242 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bcc142e90 0x7f2bcc15b450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2bc000c7e0 tx=0x7f2bc0007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:08.123 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.118+0000 7f2bc9ffb700 1 -- 172.21.15.27:0/2524561242 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc0005c00 con 0x7f2bcc142e90 2023-10-06T01:41:08.123 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.118+0000 7f2bc9ffb700 1 -- 172.21.15.27:0/2524561242 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bc0005d60 con 0x7f2bcc142e90 2023-10-06T01:41:08.123 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.118+0000 7f2bd2a2d700 1 -- 172.21.15.27:0/2524561242 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bcc15c150 con 0x7f2bcc142e90 2023-10-06T01:41:08.124 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.118+0000 7f2bc9ffb700 1 -- 172.21.15.27:0/2524561242 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc0007300 con 0x7f2bcc142e90 2023-10-06T01:41:08.124 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.118+0000 7f2bd2a2d700 1 -- 172.21.15.27:0/2524561242 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bcc15ac50 con 0x7f2bcc142e90 2023-10-06T01:41:08.124 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.119+0000 7f2bd2a2d700 1 -- 172.21.15.27:0/2524561242 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bcc04c3c0 con 0x7f2bcc142e90 2023-10-06T01:41:08.124 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.120+0000 7f2bc9ffb700 1 -- 172.21.15.27:0/2524561242 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) v1 ==== 84684+0+0 (secure 0 0 0) 0x7f2bc001b3b0 con 0x7f2bcc142e90 2023-10-06T01:41:08.124 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.120+0000 7f2bc9ffb700 1 --2- 172.21.15.27:0/2524561242 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2bb8066ea0 0x7f2bb8069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:08.124 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.120+0000 7f2bc9ffb700 1 -- 172.21.15.27:0/2524561242 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f2bc0083690 con 0x7f2bcc142e90 2023-10-06T01:41:08.124 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.120+0000 7f2bd1a2b700 1 --2- 172.21.15.27:0/2524561242 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2bb8066ea0 0x7f2bb8069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:08.124 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.120+0000 7f2bd1a2b700 1 --2- 172.21.15.27:0/2524561242 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2bb8066ea0 0x7f2bb8069350 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc0b34e0 tx=0x7f2bc40091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:08.125 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.125+0000 7f2bc9ffb700 1 -- 172.21.15.27:0/2524561242 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2bc0050bd0 con 0x7f2bcc142e90 2023-10-06T01:41:08.319 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.318+0000 7f2bd2a2d700 1 -- 172.21.15.27:0/2524561242 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2bcc15b100 con 0x7f2bcc142e90 2023-10-06T01:41:08.319 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.318+0000 7f2bc9ffb700 1 -- 172.21.15.27:0/2524561242 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2bc00170a0 con 0x7f2bcc142e90 2023-10-06T01:41:08.320 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2023-10-06T01:41:08.322 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.320+0000 7f2bd2a2d700 1 -- 172.21.15.27:0/2524561242 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2bb8066ea0 msgr2=0x7f2bb8069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:08.323 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.320+0000 7f2bd2a2d700 1 --2- 172.21.15.27:0/2524561242 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2bb8066ea0 0x7f2bb8069350 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc0b34e0 tx=0x7f2bc40091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:08.323 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.320+0000 7f2bd2a2d700 1 -- 172.21.15.27:0/2524561242 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bcc142e90 msgr2=0x7f2bcc15b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:08.323 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.320+0000 7f2bd2a2d700 1 --2- 172.21.15.27:0/2524561242 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bcc142e90 0x7f2bcc15b450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2bc000c7e0 tx=0x7f2bc0007d90 comp rx=0 tx=0).stop 2023-10-06T01:41:08.323 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.320+0000 7f2bd2a2d700 1 -- 172.21.15.27:0/2524561242 shutdown_connections 2023-10-06T01:41:08.323 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.320+0000 7f2bd2a2d700 1 --2- 172.21.15.27:0/2524561242 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2bb8066ea0 0x7f2bb8069350 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:08.323 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.320+0000 7f2bd2a2d700 1 --2- 172.21.15.27:0/2524561242 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2bcc069d50 0x7f2bcc13f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:08.323 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.320+0000 7f2bd2a2d700 1 --2- 172.21.15.27:0/2524561242 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2bcc15d9a0 0x7f2bcc16aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:08.323 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.320+0000 7f2bd2a2d700 1 --2- 172.21.15.27:0/2524561242 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2bcc142e90 0x7f2bcc15b450 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:08.324 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.320+0000 7f2bd2a2d700 1 -- 172.21.15.27:0/2524561242 >> 172.21.15.27:0/2524561242 conn(0x7f2bcc0ccfe0 msgr2=0x7f2bcc1407c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:08.324 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.320+0000 7f2bd2a2d700 1 -- 172.21.15.27:0/2524561242 shutdown_connections 2023-10-06T01:41:08.324 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.320+0000 7f2bd2a2d700 1 -- 172.21.15.27:0/2524561242 wait complete. 2023-10-06T01:41:08.333 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T01:41:08.455 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.454+0000 7f957b28e700 1 Processor -- start 2023-10-06T01:41:08.455 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.454+0000 7f957b28e700 1 -- start start 2023-10-06T01:41:08.458 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.456+0000 7f957b28e700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9574140ff0 0x7f95740aef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:08.458 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.456+0000 7f957b28e700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f95740af4b0 0x7f95740b14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:08.458 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.456+0000 7f957b28e700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95740af910 con 0x7f9574140c20 2023-10-06T01:41:08.458 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.456+0000 7f957b28e700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95740cd150 con 0x7f9574140ff0 2023-10-06T01:41:08.458 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.456+0000 7f957b28e700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95740cce70 con 0x7f95740af4b0 2023-10-06T01:41:08.458 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.456+0000 7f9578829700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9574140ff0 0x7f95740aef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:08.458 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.456+0000 7f9578829700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9574140ff0 0x7f95740aef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40240/0 (socket says 172.21.15.27:40240) 2023-10-06T01:41:08.459 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.456+0000 7f9578829700 1 -- 172.21.15.27:0/1075785864 learned_addr learned my addr 172.21.15.27:0/1075785864 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:08.459 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.456+0000 7f957982b700 1 --2- 172.21.15.27:0/1075785864 >> v2:172.21.15.124:3300/0 conn(0x7f95740af4b0 0x7f95740b14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:08.459 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.456+0000 7f9578829700 1 -- 172.21.15.27:0/1075785864 >> v2:172.21.15.124:3300/0 conn(0x7f95740af4b0 msgr2=0x7f95740b14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:08.459 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.456+0000 7f9578829700 1 --2- 172.21.15.27:0/1075785864 >> v2:172.21.15.124:3300/0 conn(0x7f95740af4b0 0x7f95740b14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:08.459 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.456+0000 7f9578829700 1 -- 172.21.15.27:0/1075785864 >> v1:172.21.15.124:6789/0 conn(0x7f9574140c20 legacy=0x7f95740cc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:08.459 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.456+0000 7f9578829700 1 -- 172.21.15.27:0/1075785864 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95740b0a10 con 0x7f9574140ff0 2023-10-06T01:41:08.459 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.456+0000 7f9578829700 1 --2- 172.21.15.27:0/1075785864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9574140ff0 0x7f95740aef70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f95740ca110 tx=0x7f9568009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91db0273bb9272b0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:08.459 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.456+0000 7f9573fff700 1 -- 172.21.15.27:0/1075785864 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9568009e70 con 0x7f9574140ff0 2023-10-06T01:41:08.459 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.456+0000 7f9573fff700 1 -- 172.21.15.27:0/1075785864 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f956800b860 con 0x7f9574140ff0 2023-10-06T01:41:08.459 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.456+0000 7f9573fff700 1 -- 172.21.15.27:0/1075785864 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956800bc00 con 0x7f9574140ff0 2023-10-06T01:41:08.459 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.457+0000 7f957b28e700 1 -- 172.21.15.27:0/1075785864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9574140ff0 msgr2=0x7f95740aef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:08.459 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.457+0000 7f957b28e700 1 --2- 172.21.15.27:0/1075785864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9574140ff0 0x7f95740aef70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f95740ca110 tx=0x7f9568009250 comp rx=0 tx=0).stop 2023-10-06T01:41:08.460 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.457+0000 7f957b28e700 1 -- 172.21.15.27:0/1075785864 shutdown_connections 2023-10-06T01:41:08.460 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.457+0000 7f957b28e700 1 --2- 172.21.15.27:0/1075785864 >> v2:172.21.15.124:3300/0 conn(0x7f95740af4b0 0x7f95740b14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:08.460 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.457+0000 7f957b28e700 1 --2- 172.21.15.27:0/1075785864 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9574140ff0 0x7f95740aef70 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:08.460 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.457+0000 7f957b28e700 1 -- 172.21.15.27:0/1075785864 >> 172.21.15.27:0/1075785864 conn(0x7f957406e6b0 msgr2=0x7f957406afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:08.460 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.457+0000 7f957b28e700 1 -- 172.21.15.27:0/1075785864 shutdown_connections 2023-10-06T01:41:08.460 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.457+0000 7f957b28e700 1 -- 172.21.15.27:0/1075785864 wait complete. 2023-10-06T01:41:08.463 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.458+0000 7f957b28e700 1 Processor -- start 2023-10-06T01:41:08.463 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.458+0000 7f957b28e700 1 -- start start 2023-10-06T01:41:08.463 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.459+0000 7f957b28e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95740af4b0 0x7f95740a6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:08.463 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.459+0000 7f957b28e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9574140c20 0x7f9574158ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:08.463 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.459+0000 7f957b28e700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9574140ff0 0x7f95741591e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:08.463 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.459+0000 7f957b28e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95740a5c40 con 0x7f95740af4b0 2023-10-06T01:41:08.463 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.459+0000 7f957b28e700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95740a5f00 con 0x7f9574140ff0 2023-10-06T01:41:08.463 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.459+0000 7f957b28e700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95740a61e0 con 0x7f9574140c20 2023-10-06T01:41:08.464 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.459+0000 7f957902a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95740af4b0 0x7f95740a6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:08.464 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.459+0000 7f957902a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95740af4b0 0x7f95740a6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40254/0 (socket says 172.21.15.27:40254) 2023-10-06T01:41:08.464 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.459+0000 7f957902a700 1 -- 172.21.15.27:0/1722513153 learned_addr learned my addr 172.21.15.27:0/1722513153 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:08.464 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.459+0000 7f957982b700 1 --2- 172.21.15.27:0/1722513153 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9574140ff0 0x7f95741591e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:08.464 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.459+0000 7f9578829700 1 --2- 172.21.15.27:0/1722513153 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9574140c20 0x7f9574158ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:08.464 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.459+0000 7f957902a700 1 -- 172.21.15.27:0/1722513153 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9574140c20 msgr2=0x7f9574158ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:08.464 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.460+0000 7f957902a700 1 --2- 172.21.15.27:0/1722513153 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9574140c20 0x7f9574158ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:08.464 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.460+0000 7f957902a700 1 -- 172.21.15.27:0/1722513153 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9574140ff0 msgr2=0x7f95741591e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:08.465 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.460+0000 7f957902a700 1 --2- 172.21.15.27:0/1722513153 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9574140ff0 0x7f95741591e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:08.465 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.460+0000 7f957902a700 1 -- 172.21.15.27:0/1722513153 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95680094f0 con 0x7f95740af4b0 2023-10-06T01:41:08.465 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.460+0000 7f957902a700 1 --2- 172.21.15.27:0/1722513153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95740af4b0 0x7f95740a6cc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f956c00c590 tx=0x7f956c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:08.465 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.460+0000 7f95717fa700 1 -- 172.21.15.27:0/1722513153 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956c011590 con 0x7f95740af4b0 2023-10-06T01:41:08.465 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.460+0000 7f95717fa700 1 -- 172.21.15.27:0/1722513153 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f956c011710 con 0x7f95740af4b0 2023-10-06T01:41:08.466 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.460+0000 7f957b28e700 1 -- 172.21.15.27:0/1722513153 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95740a64c0 con 0x7f95740af4b0 2023-10-06T01:41:08.466 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.460+0000 7f95717fa700 1 -- 172.21.15.27:0/1722513153 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956c011ad0 con 0x7f95740af4b0 2023-10-06T01:41:08.466 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.460+0000 7f957b28e700 1 -- 172.21.15.27:0/1722513153 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f957413e5b0 con 0x7f95740af4b0 2023-10-06T01:41:08.466 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.461+0000 7f957b28e700 1 -- 172.21.15.27:0/1722513153 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95740a69e0 con 0x7f95740af4b0 2023-10-06T01:41:08.466 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.462+0000 7f95717fa700 1 -- 172.21.15.27:0/1722513153 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) v1 ==== 84684+0+0 (secure 0 0 0) 0x7f956c01b3b0 con 0x7f95740af4b0 2023-10-06T01:41:08.466 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.462+0000 7f95717fa700 1 --2- 172.21.15.27:0/1722513153 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9560066ea0 0x7f9560069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:08.467 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.462+0000 7f95717fa700 1 -- 172.21.15.27:0/1722513153 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f956c083520 con 0x7f95740af4b0 2023-10-06T01:41:08.467 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.462+0000 7f9578829700 1 --2- 172.21.15.27:0/1722513153 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9560066ea0 0x7f9560069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:08.467 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.462+0000 7f9578829700 1 --2- 172.21.15.27:0/1722513153 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9560066ea0 0x7f9560069350 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f95740ca110 tx=0x7f9568017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:08.467 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.467+0000 7f95717fa700 1 -- 172.21.15.27:0/1722513153 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f956c050a60 con 0x7f95740af4b0 2023-10-06T01:41:08.704 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.704+0000 7f957b28e700 1 -- 172.21.15.27:0/1722513153 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95740af910 con 0x7f95740af4b0 2023-10-06T01:41:08.705 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.704+0000 7f95717fa700 1 -- 172.21.15.27:0/1722513153 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3527 (secure 0 0 0) 0x7f95740af910 con 0x7f95740af4b0 2023-10-06T01:41:08.705 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:08.705 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4261,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6837/2699906167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6836","nonce":2699906167},{"type":"v1","addr":"172.21.15.124:6837","nonce":2699906167}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/313372271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":313372271},{"type":"v1","addr":"172.21.15.27:6834","nonce":313372271}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4287,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/4257743842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4257743842},{"type":"v1","addr":"172.21.15.27:6835","nonce":4257743842}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4305,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6835/3803994994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":3803994994},{"type":"v1","addr":"172.21.15.124:6835","nonce":3803994994}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-10-06T01:41:08.707 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.706+0000 7f957b28e700 1 -- 172.21.15.27:0/1722513153 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9560066ea0 msgr2=0x7f9560069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:08.707 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.706+0000 7f957b28e700 1 --2- 172.21.15.27:0/1722513153 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9560066ea0 0x7f9560069350 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f95740ca110 tx=0x7f9568017000 comp rx=0 tx=0).stop 2023-10-06T01:41:08.707 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.706+0000 7f957b28e700 1 -- 172.21.15.27:0/1722513153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95740af4b0 msgr2=0x7f95740a6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:08.707 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.706+0000 7f957b28e700 1 --2- 172.21.15.27:0/1722513153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95740af4b0 0x7f95740a6cc0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f956c00c590 tx=0x7f956c00c8a0 comp rx=0 tx=0).stop 2023-10-06T01:41:08.707 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.707+0000 7f957b28e700 1 -- 172.21.15.27:0/1722513153 shutdown_connections 2023-10-06T01:41:08.707 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.707+0000 7f957b28e700 1 --2- 172.21.15.27:0/1722513153 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9560066ea0 0x7f9560069350 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:08.707 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.707+0000 7f957b28e700 1 --2- 172.21.15.27:0/1722513153 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9574140ff0 0x7f95741591e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:08.708 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.707+0000 7f957b28e700 1 --2- 172.21.15.27:0/1722513153 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f95740af4b0 0x7f95740a6cc0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:08.708 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.707+0000 7f957b28e700 1 --2- 172.21.15.27:0/1722513153 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9574140c20 0x7f9574158ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:08.708 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.707+0000 7f957b28e700 1 -- 172.21.15.27:0/1722513153 >> 172.21.15.27:0/1722513153 conn(0x7f957406e6b0 msgr2=0x7f957413e810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:08.708 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.707+0000 7f957b28e700 1 -- 172.21.15.27:0/1722513153 shutdown_connections 2023-10-06T01:41:08.708 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.707+0000 7f957b28e700 1 -- 172.21.15.27:0/1722513153 wait complete. 2023-10-06T01:41:08.708 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 3 2023-10-06T01:41:08.720 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T01:41:08.838 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.838+0000 7f02c1632700 1 Processor -- start 2023-10-06T01:41:08.839 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.838+0000 7f02c1632700 1 -- start start 2023-10-06T01:41:08.841 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.839+0000 7f02c1632700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f02bc15d560 0x7f02bc0d0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:08.841 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.839+0000 7f02c1632700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f02bc15d930 0x7f02bc0b1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:08.841 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.839+0000 7f02c1632700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02bc0b2ad0 con 0x7f02bc1591b0 2023-10-06T01:41:08.841 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.839+0000 7f02c1632700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02bc0b4160 con 0x7f02bc15d560 2023-10-06T01:41:08.841 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.839+0000 7f02c1632700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02bc0b1f80 con 0x7f02bc15d930 2023-10-06T01:41:08.841 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.839+0000 7f02baffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f02bc15d560 0x7f02bc0d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:08.841 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.839+0000 7f02baffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f02bc15d560 0x7f02bc0d0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40262/0 (socket says 172.21.15.27:40262) 2023-10-06T01:41:08.841 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.839+0000 7f02baffd700 1 -- 172.21.15.27:0/2680446483 learned_addr learned my addr 172.21.15.27:0/2680446483 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:08.842 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.839+0000 7f02ba7fc700 1 --2- 172.21.15.27:0/2680446483 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f02bc15d930 0x7f02bc0b1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:08.842 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.839+0000 7f02baffd700 1 -- 172.21.15.27:0/2680446483 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f02bc15d930 msgr2=0x7f02bc0b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:08.842 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.839+0000 7f02baffd700 1 --2- 172.21.15.27:0/2680446483 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f02bc15d930 0x7f02bc0b1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:08.842 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.839+0000 7f02baffd700 1 -- 172.21.15.27:0/2680446483 >> v1:172.21.15.124:6789/0 conn(0x7f02bc1591b0 legacy=0x7f02bc159630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:08.842 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.839+0000 7f02baffd700 1 -- 172.21.15.27:0/2680446483 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02bc0b2230 con 0x7f02bc15d560 2023-10-06T01:41:08.842 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.839+0000 7f02baffd700 1 --2- 172.21.15.27:0/2680446483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02bc15d560 0x7f02bc0d0030 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f02b40099b0 tx=0x7f02b4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e720770a6c66124 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:08.842 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.840+0000 7f02b9ffb700 1 -- 172.21.15.27:0/2680446483 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b400b820 con 0x7f02bc15d560 2023-10-06T01:41:08.842 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.840+0000 7f02b9ffb700 1 -- 172.21.15.27:0/2680446483 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02b400b980 con 0x7f02bc15d560 2023-10-06T01:41:08.842 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.840+0000 7f02b9ffb700 1 -- 172.21.15.27:0/2680446483 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b400bd20 con 0x7f02bc15d560 2023-10-06T01:41:08.842 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.840+0000 7f02c1632700 1 -- 172.21.15.27:0/2680446483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02bc15d560 msgr2=0x7f02bc0d0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:08.842 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.840+0000 7f02c1632700 1 --2- 172.21.15.27:0/2680446483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02bc15d560 0x7f02bc0d0030 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f02b40099b0 tx=0x7f02b4009250 comp rx=0 tx=0).stop 2023-10-06T01:41:08.842 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.840+0000 7f02c1632700 1 -- 172.21.15.27:0/2680446483 shutdown_connections 2023-10-06T01:41:08.843 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.840+0000 7f02c1632700 1 --2- 172.21.15.27:0/2680446483 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02bc15d560 0x7f02bc0d0030 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:08.843 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.840+0000 7f02c1632700 1 --2- 172.21.15.27:0/2680446483 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f02bc15d930 0x7f02bc0b1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:08.843 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.840+0000 7f02c1632700 1 -- 172.21.15.27:0/2680446483 >> 172.21.15.27:0/2680446483 conn(0x7f02bc0d1180 msgr2=0x7f02bc069640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:08.843 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.840+0000 7f02c1632700 1 -- 172.21.15.27:0/2680446483 shutdown_connections 2023-10-06T01:41:08.843 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.840+0000 7f02c1632700 1 -- 172.21.15.27:0/2680446483 wait complete. 2023-10-06T01:41:08.845 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.841+0000 7f02c1632700 1 Processor -- start 2023-10-06T01:41:08.845 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.841+0000 7f02c1632700 1 -- start start 2023-10-06T01:41:08.846 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.842+0000 7f02c1632700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02bc1591b0 0x7f02bc14b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:08.846 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.842+0000 7f02c1632700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f02bc15d560 0x7f02bc14bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:08.846 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.842+0000 7f02c1632700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02bc15d930 0x7f02bc14ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:08.846 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.842+0000 7f02c1632700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02bc15bc10 con 0x7f02bc1591b0 2023-10-06T01:41:08.846 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.842+0000 7f02c1632700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02bc15b070 con 0x7f02bc15d930 2023-10-06T01:41:08.846 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.842+0000 7f02c1632700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02bc15b2e0 con 0x7f02bc15d560 2023-10-06T01:41:08.846 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.842+0000 7f02baffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02bc1591b0 0x7f02bc14b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:08.846 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.842+0000 7f02baffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02bc1591b0 0x7f02bc14b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40266/0 (socket says 172.21.15.27:40266) 2023-10-06T01:41:08.846 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.842+0000 7f02baffd700 1 -- 172.21.15.27:0/2366741978 learned_addr learned my addr 172.21.15.27:0/2366741978 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:08.846 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.842+0000 7f02ba7fc700 1 --2- 172.21.15.27:0/2366741978 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f02bc15d560 0x7f02bc14bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:08.846 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.842+0000 7f02bb7fe700 1 --2- 172.21.15.27:0/2366741978 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02bc15d930 0x7f02bc14ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:08.847 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.842+0000 7f02baffd700 1 -- 172.21.15.27:0/2366741978 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f02bc15d560 msgr2=0x7f02bc14bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:08.847 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.842+0000 7f02baffd700 1 --2- 172.21.15.27:0/2366741978 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f02bc15d560 0x7f02bc14bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:08.847 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.842+0000 7f02baffd700 1 -- 172.21.15.27:0/2366741978 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02bc15d930 msgr2=0x7f02bc14ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:08.847 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.842+0000 7f02baffd700 1 --2- 172.21.15.27:0/2366741978 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02bc15d930 0x7f02bc14ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:08.847 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.842+0000 7f02baffd700 1 -- 172.21.15.27:0/2366741978 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02b40094f0 con 0x7f02bc1591b0 2023-10-06T01:41:08.847 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.843+0000 7f02bb7fe700 1 --2- 172.21.15.27:0/2366741978 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02bc15d930 0x7f02bc14ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:41:08.847 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.843+0000 7f02baffd700 1 --2- 172.21.15.27:0/2366741978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02bc1591b0 0x7f02bc14b4b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f02b40094c0 tx=0x7f02b4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:08.847 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.843+0000 7f02b2ffd700 1 -- 172.21.15.27:0/2366741978 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b401b470 con 0x7f02bc1591b0 2023-10-06T01:41:08.847 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.843+0000 7f02b2ffd700 1 -- 172.21.15.27:0/2366741978 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02b401b5d0 con 0x7f02bc1591b0 2023-10-06T01:41:08.847 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.843+0000 7f02c1632700 1 -- 172.21.15.27:0/2366741978 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02bc15b5c0 con 0x7f02bc1591b0 2023-10-06T01:41:08.847 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.843+0000 7f02b2ffd700 1 -- 172.21.15.27:0/2366741978 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02b401b990 con 0x7f02bc1591b0 2023-10-06T01:41:08.847 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.843+0000 7f02c1632700 1 -- 172.21.15.27:0/2366741978 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02bc14b9f0 con 0x7f02bc1591b0 2023-10-06T01:41:08.848 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.844+0000 7f02c1632700 1 -- 172.21.15.27:0/2366741978 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02bc064050 con 0x7f02bc1591b0 2023-10-06T01:41:08.848 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.844+0000 7f02b2ffd700 1 -- 172.21.15.27:0/2366741978 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 4) v1 ==== 84684+0+0 (secure 0 0 0) 0x7f02b401c450 con 0x7f02bc1591b0 2023-10-06T01:41:08.848 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.844+0000 7f02b2ffd700 1 --2- 172.21.15.27:0/2366741978 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f029c066ea0 0x7f029c069350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:08.848 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.845+0000 7f02b2ffd700 1 -- 172.21.15.27:0/2366741978 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f02b4085f80 con 0x7f02bc1591b0 2023-10-06T01:41:08.848 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.845+0000 7f02ba7fc700 1 --2- 172.21.15.27:0/2366741978 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f029c066ea0 0x7f029c069350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:08.848 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.845+0000 7f02ba7fc700 1 --2- 172.21.15.27:0/2366741978 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f029c066ea0 0x7f029c069350 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f02a4006fd0 tx=0x7f02a4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:08.849 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.849+0000 7f02b2ffd700 1 -- 172.21.15.27:0/2366741978 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f02b40534c0 con 0x7f02bc1591b0 2023-10-06T01:41:08.922 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:08.922+0000 7f02b2ffd700 1 -- 172.21.15.27:0/2366741978 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f02b4053890 con 0x7f02bc1591b0 2023-10-06T01:41:09.046 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.045+0000 7f02c1632700 1 -- 172.21.15.27:0/2366741978 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f02bc0b2510 con 0x7f02bc1591b0 2023-10-06T01:41:09.046 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.046+0000 7f02b2ffd700 1 -- 172.21.15.27:0/2366741978 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+13648 (secure 0 0 0) 0x7f02b4056f50 con 0x7f02bc1591b0 2023-10-06T01:41:09.047 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:09.047 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":16,"fsid":"20a7bff0-74f8-4f2f-b6c6-899e804bc965","created":"2023-10-06T01:40:54.474215+0000","modified":"2023-10-06T01:41:08.252495+0000","last_up_change":"2023-10-06T01:41:05.242847+0000","last_in_change":"2023-10-06T01:40:56.890960+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-10-06T01:41:05.685386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-10-06T01:41:06.092995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b902c86d-800c-40c5-b152-b545e50d02c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":90644},{"type":"v1","addr":"172.21.15.27:6801","nonce":90644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":90644},{"type":"v1","addr":"172.21.15.27:6803","nonce":90644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":90644},{"type":"v1","addr":"172.21.15.27:6807","nonce":90644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":90644},{"type":"v1","addr":"172.21.15.27:6805","nonce":90644}]},"public_addr":"172.21.15.27:6801/90644","cluster_addr":"172.21.15.27:6803/90644","heartbeat_back_addr":"172.21.15.27:6807/90644","heartbeat_front_addr":"172.21.15.27:6805/90644","state":["exists","up"]},{"osd":1,"uuid":"5b4a70e5-b06e-48cd-bfc4-7962c5d05386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":90643},{"type":"v1","addr":"172.21.15.27:6809","nonce":90643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":90643},{"type":"v1","addr":"172.21.15.27:6811","nonce":90643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":90643},{"type":"v1","addr":"172.21.15.27:6815","nonce":90643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":90643},{"type":"v1","addr":"172.21.15.27:6813","nonce":90643}]},"public_addr":"172.21.15.27:6809/90643","cluster_addr":"172.21.15.27:6811/90643","heartbeat_back_addr":"172.21.15.27:6815/90643","heartbeat_front_addr":"172.21.15.27:6813/90643","state":["exists","up"]},{"osd":2,"uuid":"22d9de46-4073-4336-9122-5227e601eca0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":90645},{"type":"v1","addr":"172.21.15.27:6817","nonce":90645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":90645},{"type":"v1","addr":"172.21.15.27:6819","nonce":90645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":90645},{"type":"v1","addr":"172.21.15.27:6823","nonce":90645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":90645},{"type":"v1","addr":"172.21.15.27:6821","nonce":90645}]},"public_addr":"172.21.15.27:6817/90645","cluster_addr":"172.21.15.27:6819/90645","heartbeat_back_addr":"172.21.15.27:6823/90645","heartbeat_front_addr":"172.21.15.27:6821/90645","state":["exists","up"]},{"osd":3,"uuid":"438bb2d3-34e2-4b7c-9734-8360c1b67650","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":90668},{"type":"v1","addr":"172.21.15.27:6825","nonce":90668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":90668},{"type":"v1","addr":"172.21.15.27:6827","nonce":90668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":90668},{"type":"v1","addr":"172.21.15.27:6831","nonce":90668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":90668},{"type":"v1","addr":"172.21.15.27:6829","nonce":90668}]},"public_addr":"172.21.15.27:6825/90668","cluster_addr":"172.21.15.27:6827/90668","heartbeat_back_addr":"172.21.15.27:6831/90668","heartbeat_front_addr":"172.21.15.27:6829/90668","state":["exists","up"]},{"osd":4,"uuid":"e1956a8e-e0f8-4e7c-acbb-40037415a354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":90863},{"type":"v1","addr":"172.21.15.124:6801","nonce":90863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":90863},{"type":"v1","addr":"172.21.15.124:6803","nonce":90863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":90863},{"type":"v1","addr":"172.21.15.124:6807","nonce":90863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":90863},{"type":"v1","addr":"172.21.15.124:6805","nonce":90863}]},"public_addr":"172.21.15.124:6801/90863","cluster_addr":"172.21.15.124:6803/90863","heartbeat_back_addr":"172.21.15.124:6807/90863","heartbeat_front_addr":"172.21.15.124:6805/90863","state":["exists","up"]},{"osd":5,"uuid":"306c59d4-846d-4b73-b1c9-051bec9e9368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":90885},{"type":"v1","addr":"172.21.15.124:6817","nonce":90885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":90885},{"type":"v1","addr":"172.21.15.124:6819","nonce":90885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":90885},{"type":"v1","addr":"172.21.15.124:6823","nonce":90885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":90885},{"type":"v1","addr":"172.21.15.124:6821","nonce":90885}]},"public_addr":"172.21.15.124:6817/90885","cluster_addr":"172.21.15.124:6819/90885","heartbeat_back_addr":"172.21.15.124:6823/90885","heartbeat_front_addr":"172.21.15.124:6821/90885","state":["exists","up"]},{"osd":6,"uuid":"15463e53-ac8d-47d1-b4f6-c0d31f29e4a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":90864},{"type":"v1","addr":"172.21.15.124:6809","nonce":90864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":90864},{"type":"v1","addr":"172.21.15.124:6811","nonce":90864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":90864},{"type":"v1","addr":"172.21.15.124:6815","nonce":90864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":90864},{"type":"v1","addr":"172.21.15.124:6813","nonce":90864}]},"public_addr":"172.21.15.124:6809/90864","cluster_addr":"172.21.15.124:6811/90864","heartbeat_back_addr":"172.21.15.124:6815/90864","heartbeat_front_addr":"172.21.15.124:6813/90864","state":["exists","up"]},{"osd":7,"uuid":"ae463a90-a9d3-498f-86fa-47a35b3fe889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":90894},{"type":"v1","addr":"172.21.15.124:6825","nonce":90894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":90894},{"type":"v1","addr":"172.21.15.124:6827","nonce":90894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":90894},{"type":"v1","addr":"172.21.15.124:6831","nonce":90894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":90894},{"type":"v1","addr":"172.21.15.124:6829","nonce":90894}]},"public_addr":"172.21.15.124:6825/90894","cluster_addr":"172.21.15.124:6827/90894","heartbeat_back_addr":"172.21.15.124:6831/90894","heartbeat_front_addr":"172.21.15.124:6829/90894","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.827704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.839951+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.032230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.050605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.987908+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.031290+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.080707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.053692+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T01:41:09.049 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.048+0000 7f02c1632700 1 -- 172.21.15.27:0/2366741978 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f029c066ea0 msgr2=0x7f029c069350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:09.049 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.048+0000 7f02c1632700 1 --2- 172.21.15.27:0/2366741978 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f029c066ea0 0x7f029c069350 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f02a4006fd0 tx=0x7f02a4008040 comp rx=0 tx=0).stop 2023-10-06T01:41:09.049 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.048+0000 7f02c1632700 1 -- 172.21.15.27:0/2366741978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02bc1591b0 msgr2=0x7f02bc14b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:09.049 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.048+0000 7f02c1632700 1 --2- 172.21.15.27:0/2366741978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02bc1591b0 0x7f02bc14b4b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f02b40094c0 tx=0x7f02b4016000 comp rx=0 tx=0).stop 2023-10-06T01:41:09.049 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.048+0000 7f02c1632700 1 -- 172.21.15.27:0/2366741978 shutdown_connections 2023-10-06T01:41:09.049 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.048+0000 7f02c1632700 1 --2- 172.21.15.27:0/2366741978 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f029c066ea0 0x7f029c069350 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:09.049 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.048+0000 7f02c1632700 1 --2- 172.21.15.27:0/2366741978 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f02bc15d930 0x7f02bc14ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:09.049 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.048+0000 7f02c1632700 1 --2- 172.21.15.27:0/2366741978 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f02bc1591b0 0x7f02bc14b4b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:09.049 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.048+0000 7f02c1632700 1 --2- 172.21.15.27:0/2366741978 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f02bc15d560 0x7f02bc14bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:09.049 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.048+0000 7f02c1632700 1 -- 172.21.15.27:0/2366741978 >> 172.21.15.27:0/2366741978 conn(0x7f02bc0d1180 msgr2=0x7f02bc0b1750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:09.049 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.048+0000 7f02c1632700 1 -- 172.21.15.27:0/2366741978 shutdown_connections 2023-10-06T01:41:09.049 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.048+0000 7f02c1632700 1 -- 172.21.15.27:0/2366741978 wait complete. 2023-10-06T01:41:09.061 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-10-06T01:41:05.685386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-10-06T01:41:06.092995+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-10-06T01:41:09.061 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-06T01:41:09.182 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.182+0000 7f5a5234c700 1 Processor -- start 2023-10-06T01:41:09.183 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.182+0000 7f5a5234c700 1 -- start start 2023-10-06T01:41:09.185 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.183+0000 7f5a5234c700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5a4c15de80 0x7f5a4c0b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:09.185 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.183+0000 7f5a5234c700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a4c069380 con 0x7f5a4c142e90 2023-10-06T01:41:09.185 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.183+0000 7f5a5234c700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a4c069f20 con 0x7f5a4c0b61d0 2023-10-06T01:41:09.185 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.183+0000 7f5a5234c700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4c068770 con 0x7f5a4c15de80 2023-10-06T01:41:09.185 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.183+0000 7f5a5134a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5a4c142e90 0x7f5a4c069810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47304/0 (socket says 172.21.15.27:47304) 2023-10-06T01:41:09.185 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.183+0000 7f5a5134a700 1 -- 172.21.15.27:0/1430661792 learned_addr learned my addr 172.21.15.27:0/1430661792 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:09.186 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.183+0000 7f5a50b49700 1 --2- 172.21.15.27:0/1430661792 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5a4c15de80 0x7f5a4c0b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:09.186 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.183+0000 7f5a4bfff700 1 -- 172.21.15.27:0/1430661792 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 454571766 0 0) 0x7f5a4c069380 con 0x7f5a4c142e90 2023-10-06T01:41:09.186 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.183+0000 7f5a4bfff700 1 -- 172.21.15.27:0/1430661792 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a34003570 con 0x7f5a4c142e90 2023-10-06T01:41:09.186 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.183+0000 7f5a4bfff700 1 -- 172.21.15.27:0/1430661792 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3385001602 0 0) 0x7f5a4c069f20 con 0x7f5a4c0b61d0 2023-10-06T01:41:09.186 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.183+0000 7f5a4bfff700 1 -- 172.21.15.27:0/1430661792 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a4c069380 con 0x7f5a4c0b61d0 2023-10-06T01:41:09.187 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.184+0000 7f5a50b49700 1 -- 172.21.15.27:0/1430661792 >> v1:172.21.15.124:6789/0 conn(0x7f5a4c0b61d0 legacy=0x7f5a4c15d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:09.187 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.184+0000 7f5a50b49700 1 -- 172.21.15.27:0/1430661792 >> v1:172.21.15.27:6789/0 conn(0x7f5a4c142e90 legacy=0x7f5a4c069810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:09.187 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.184+0000 7f5a50b49700 1 -- 172.21.15.27:0/1430661792 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a4c068a50 con 0x7f5a4c15de80 2023-10-06T01:41:09.187 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.184+0000 7f5a50b49700 1 --2- 172.21.15.27:0/1430661792 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5a4c15de80 0x7f5a4c0b5c90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5a40009910 tx=0x7f5a400091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf92c99d9d9d456f server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:09.187 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.184+0000 7f5a4bfff700 1 -- 172.21.15.27:0/1430661792 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a40011630 con 0x7f5a4c15de80 2023-10-06T01:41:09.187 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.184+0000 7f5a4bfff700 1 -- 172.21.15.27:0/1430661792 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a40011790 con 0x7f5a4c15de80 2023-10-06T01:41:09.188 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.184+0000 7f5a4bfff700 1 -- 172.21.15.27:0/1430661792 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a40011b10 con 0x7f5a4c15de80 2023-10-06T01:41:09.188 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.184+0000 7f5a5234c700 1 -- 172.21.15.27:0/1430661792 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5a4c15de80 msgr2=0x7f5a4c0b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:09.188 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.184+0000 7f5a5234c700 1 --2- 172.21.15.27:0/1430661792 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5a4c15de80 0x7f5a4c0b5c90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5a40009910 tx=0x7f5a400091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:09.188 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.185+0000 7f5a5234c700 1 -- 172.21.15.27:0/1430661792 shutdown_connections 2023-10-06T01:41:09.188 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.185+0000 7f5a5234c700 1 --2- 172.21.15.27:0/1430661792 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5a4c15de80 0x7f5a4c0b5c90 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:09.188 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.185+0000 7f5a5234c700 1 -- 172.21.15.27:0/1430661792 >> 172.21.15.27:0/1430661792 conn(0x7f5a4c0ccfe0 msgr2=0x7f5a4c0cd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:09.188 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.185+0000 7f5a5234c700 1 -- 172.21.15.27:0/1430661792 shutdown_connections 2023-10-06T01:41:09.189 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.185+0000 7f5a5234c700 1 -- 172.21.15.27:0/1430661792 wait complete. 2023-10-06T01:41:09.193 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.186+0000 7f5a5234c700 1 Processor -- start 2023-10-06T01:41:09.193 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.186+0000 7f5a5234c700 1 -- start start 2023-10-06T01:41:09.193 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.187+0000 7f5a5234c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a4c0b61d0 0x7f5a4c16a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:09.193 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.187+0000 7f5a5234c700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5a4c142e90 0x7f5a4c0cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:09.193 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.187+0000 7f5a5234c700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5a4c15de80 0x7f5a4c0cbee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:09.193 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.187+0000 7f5a5234c700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4c0c7db0 con 0x7f5a4c0b61d0 2023-10-06T01:41:09.193 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.187+0000 7f5a5234c700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4c16aa70 con 0x7f5a4c15de80 2023-10-06T01:41:09.193 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.187+0000 7f5a5234c700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a4c0c71a0 con 0x7f5a4c142e90 2023-10-06T01:41:09.193 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.187+0000 7f5a5134a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a4c0b61d0 0x7f5a4c16a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:09.193 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.187+0000 7f5a5134a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a4c0b61d0 0x7f5a4c16a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40272/0 (socket says 172.21.15.27:40272) 2023-10-06T01:41:09.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.187+0000 7f5a5134a700 1 -- 172.21.15.27:0/855514612 learned_addr learned my addr 172.21.15.27:0/855514612 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:09.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.187+0000 7f5a50b49700 1 --2- 172.21.15.27:0/855514612 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5a4c142e90 0x7f5a4c0cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:09.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.187+0000 7f5a51b4b700 1 --2- 172.21.15.27:0/855514612 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5a4c15de80 0x7f5a4c0cbee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:09.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.187+0000 7f5a5134a700 1 -- 172.21.15.27:0/855514612 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5a4c142e90 msgr2=0x7f5a4c0cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:09.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.187+0000 7f5a5134a700 1 --2- 172.21.15.27:0/855514612 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5a4c142e90 0x7f5a4c0cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:09.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.187+0000 7f5a5134a700 1 -- 172.21.15.27:0/855514612 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5a4c15de80 msgr2=0x7f5a4c0cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:09.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.187+0000 7f5a5134a700 1 --2- 172.21.15.27:0/855514612 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5a4c15de80 0x7f5a4c0cbee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:09.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.187+0000 7f5a5134a700 1 -- 172.21.15.27:0/855514612 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a40009420 con 0x7f5a4c0b61d0 2023-10-06T01:41:09.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.187+0000 7f5a5134a700 1 --2- 172.21.15.27:0/855514612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a4c0b61d0 0x7f5a4c16a530 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5a44002250 tx=0x7f5a44001bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:09.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.187+0000 7f5a497fa700 1 -- 172.21.15.27:0/855514612 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a4400ce90 con 0x7f5a4c0b61d0 2023-10-06T01:41:09.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.187+0000 7f5a497fa700 1 -- 172.21.15.27:0/855514612 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a4400aa80 con 0x7f5a4c0b61d0 2023-10-06T01:41:09.195 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.187+0000 7f5a5234c700 1 -- 172.21.15.27:0/855514612 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a4c0c7480 con 0x7f5a4c0b61d0 2023-10-06T01:41:09.195 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.187+0000 7f5a497fa700 1 -- 172.21.15.27:0/855514612 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a4400ae20 con 0x7f5a4c0b61d0 2023-10-06T01:41:09.195 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.187+0000 7f5a5234c700 1 -- 172.21.15.27:0/855514612 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a4c0c7a40 con 0x7f5a4c0b61d0 2023-10-06T01:41:09.195 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.188+0000 7f5a5234c700 1 -- 172.21.15.27:0/855514612 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a4c04c3c0 con 0x7f5a4c0b61d0 2023-10-06T01:41:09.195 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.189+0000 7f5a497fa700 1 -- 172.21.15.27:0/855514612 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5a44007300 con 0x7f5a4c0b61d0 2023-10-06T01:41:09.195 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.189+0000 7f5a497fa700 1 --2- 172.21.15.27:0/855514612 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5a38066f30 0x7f5a380693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:09.195 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.189+0000 7f5a497fa700 1 -- 172.21.15.27:0/855514612 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f5a44084790 con 0x7f5a4c0b61d0 2023-10-06T01:41:09.195 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.189+0000 7f5a50b49700 1 --2- 172.21.15.27:0/855514612 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5a38066f30 0x7f5a380693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:09.195 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.190+0000 7f5a50b49700 1 --2- 172.21.15.27:0/855514612 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5a38066f30 0x7f5a380693e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c15e280 tx=0x7f5a40017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:09.196 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.193+0000 7f5a497fa700 1 -- 172.21.15.27:0/855514612 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5a44055700 con 0x7f5a4c0b61d0 2023-10-06T01:41:09.385 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.384+0000 7f5a5234c700 1 -- 172.21.15.27:0/855514612 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f5a4c068d30 con 0x7f5a4c0b61d0 2023-10-06T01:41:09.385 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.385+0000 7f5a497fa700 1 -- 172.21.15.27:0/855514612 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v16) v1 ==== 92+0+10 (secure 0 0 0) 0x7f5a44055290 con 0x7f5a4c0b61d0 2023-10-06T01:41:09.385 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 8 2023-10-06T01:41:09.387 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.387+0000 7f5a5234c700 1 -- 172.21.15.27:0/855514612 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5a38066f30 msgr2=0x7f5a380693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:09.387 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.387+0000 7f5a5234c700 1 --2- 172.21.15.27:0/855514612 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5a38066f30 0x7f5a380693e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5a4c15e280 tx=0x7f5a40017000 comp rx=0 tx=0).stop 2023-10-06T01:41:09.387 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.387+0000 7f5a5234c700 1 -- 172.21.15.27:0/855514612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a4c0b61d0 msgr2=0x7f5a4c16a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:09.387 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.387+0000 7f5a5234c700 1 --2- 172.21.15.27:0/855514612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a4c0b61d0 0x7f5a4c16a530 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5a44002250 tx=0x7f5a44001bb0 comp rx=0 tx=0).stop 2023-10-06T01:41:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.387+0000 7f5a5234c700 1 -- 172.21.15.27:0/855514612 shutdown_connections 2023-10-06T01:41:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.387+0000 7f5a5234c700 1 --2- 172.21.15.27:0/855514612 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5a38066f30 0x7f5a380693e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.387+0000 7f5a5234c700 1 --2- 172.21.15.27:0/855514612 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5a4c15de80 0x7f5a4c0cbee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.387+0000 7f5a5234c700 1 --2- 172.21.15.27:0/855514612 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5a4c0b61d0 0x7f5a4c16a530 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.387+0000 7f5a5234c700 1 --2- 172.21.15.27:0/855514612 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5a4c142e90 0x7f5a4c0cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.387+0000 7f5a5234c700 1 -- 172.21.15.27:0/855514612 >> 172.21.15.27:0/855514612 conn(0x7f5a4c0ccfe0 msgr2=0x7f5a4c0b1650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.387+0000 7f5a5234c700 1 -- 172.21.15.27:0/855514612 shutdown_connections 2023-10-06T01:41:09.388 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.387+0000 7f5a5234c700 1 -- 172.21.15.27:0/855514612 wait complete. 2023-10-06T01:41:09.399 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-06T01:41:09.521 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.520+0000 7f631a9d2700 1 Processor -- start 2023-10-06T01:41:09.521 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.521+0000 7f631a9d2700 1 -- start start 2023-10-06T01:41:09.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.521+0000 7f631a9d2700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f631415cb10 0x7f63140af780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:09.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.521+0000 7f631a9d2700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f63140b1690 0x7f63140b1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:09.525 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.521+0000 7f631a9d2700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63140698e0 con 0x7f63140b4000 2023-10-06T01:41:09.525 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.521+0000 7f631a9d2700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63140b2020 con 0x7f63140b1690 2023-10-06T01:41:09.525 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.521+0000 7f631a9d2700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f631415a260 con 0x7f631415cb10 2023-10-06T01:41:09.525 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.522+0000 7f6313fff700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f631415cb10 0x7f63140af780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:09.525 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.522+0000 7f6318f6f700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f63140b1690 0x7f63140b1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:09.525 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.522+0000 7f6313fff700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f631415cb10 0x7f63140af780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.27:50732/0 (socket says 172.21.15.27:50732) 2023-10-06T01:41:09.525 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.522+0000 7f6318f6f700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f63140b1690 0x7f63140b1ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.27:38002/0 (socket says 172.21.15.27:38002) 2023-10-06T01:41:09.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.522+0000 7f6313fff700 1 -- 172.21.15.27:0/1188283611 learned_addr learned my addr 172.21.15.27:0/1188283611 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:09.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.522+0000 7f63137fe700 1 --1- 172.21.15.27:0/1188283611 >> v1:172.21.15.124:6789/0 conn(0x7f63140b4000 0x7f6314159b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.27:38476/0 (socket says 172.21.15.27:38476) 2023-10-06T01:41:09.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.522+0000 7f6313fff700 1 -- 172.21.15.27:0/1188283611 >> v2:172.21.15.124:3300/0 conn(0x7f63140b1690 msgr2=0x7f63140b1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:09.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.522+0000 7f6313fff700 1 --2- 172.21.15.27:0/1188283611 >> v2:172.21.15.124:3300/0 conn(0x7f63140b1690 0x7f63140b1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:09.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.522+0000 7f6313fff700 1 -- 172.21.15.27:0/1188283611 >> v1:172.21.15.124:6789/0 conn(0x7f63140b4000 legacy=0x7f6314159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:09.527 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.522+0000 7f6313fff700 1 -- 172.21.15.27:0/1188283611 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6314068fd0 con 0x7f631415cb10 2023-10-06T01:41:09.527 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.522+0000 7f6318f6f700 1 --2- 172.21.15.27:0/1188283611 >> v2:172.21.15.124:3300/0 conn(0x7f63140b1690 0x7f63140b1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:41:09.527 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.522+0000 7f6313fff700 1 --2- 172.21.15.27:0/1188283611 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f631415cb10 0x7f63140af780 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f63080099b0 tx=0x7f6308009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce1f8db2baa4b104 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:09.527 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.522+0000 7f6312ffd700 1 -- 172.21.15.27:0/1188283611 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6308009ea0 con 0x7f631415cb10 2023-10-06T01:41:09.527 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.522+0000 7f6312ffd700 1 -- 172.21.15.27:0/1188283611 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f630800b860 con 0x7f631415cb10 2023-10-06T01:41:09.527 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.522+0000 7f6312ffd700 1 -- 172.21.15.27:0/1188283611 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630800bc00 con 0x7f631415cb10 2023-10-06T01:41:09.528 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.523+0000 7f631a9d2700 1 -- 172.21.15.27:0/1188283611 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f631415cb10 msgr2=0x7f63140af780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:09.528 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.523+0000 7f631a9d2700 1 --2- 172.21.15.27:0/1188283611 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f631415cb10 0x7f63140af780 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f63080099b0 tx=0x7f6308009250 comp rx=0 tx=0).stop 2023-10-06T01:41:09.528 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.523+0000 7f631a9d2700 1 -- 172.21.15.27:0/1188283611 shutdown_connections 2023-10-06T01:41:09.528 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.523+0000 7f631a9d2700 1 --2- 172.21.15.27:0/1188283611 >> v2:172.21.15.124:3300/0 conn(0x7f63140b1690 0x7f63140b1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:09.528 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.523+0000 7f631a9d2700 1 --2- 172.21.15.27:0/1188283611 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f631415cb10 0x7f63140af780 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:09.528 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.523+0000 7f631a9d2700 1 -- 172.21.15.27:0/1188283611 >> 172.21.15.27:0/1188283611 conn(0x7f631406d890 msgr2=0x7f631406dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:09.528 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.523+0000 7f631a9d2700 1 -- 172.21.15.27:0/1188283611 shutdown_connections 2023-10-06T01:41:09.529 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.523+0000 7f631a9d2700 1 -- 172.21.15.27:0/1188283611 wait complete. 2023-10-06T01:41:09.529 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.524+0000 7f631a9d2700 1 Processor -- start 2023-10-06T01:41:09.529 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.524+0000 7f631a9d2700 1 -- start start 2023-10-06T01:41:09.533 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.525+0000 7f631a9d2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63140b1690 0x7f6314157460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:09.533 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.525+0000 7f631a9d2700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63140b4000 0x7f63140a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:09.533 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.525+0000 7f631a9d2700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f631415cb10 0x7f63140a60c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:09.533 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.525+0000 7f631a9d2700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63140cdc90 con 0x7f63140b1690 2023-10-06T01:41:09.533 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.525+0000 7f631a9d2700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63140cd0c0 con 0x7f63140b4000 2023-10-06T01:41:09.533 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.525+0000 7f631a9d2700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63140cd360 con 0x7f631415cb10 2023-10-06T01:41:09.533 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.525+0000 7f6313fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63140b1690 0x7f6314157460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:09.533 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.525+0000 7f6313fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63140b1690 0x7f6314157460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40278/0 (socket says 172.21.15.27:40278) 2023-10-06T01:41:09.533 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.525+0000 7f6313fff700 1 -- 172.21.15.27:0/2513109965 learned_addr learned my addr 172.21.15.27:0/2513109965 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:09.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.525+0000 7f63137fe700 1 --2- 172.21.15.27:0/2513109965 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63140b4000 0x7f63140a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:09.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.525+0000 7f6318f6f700 1 --2- 172.21.15.27:0/2513109965 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f631415cb10 0x7f63140a60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:09.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.525+0000 7f6313fff700 1 -- 172.21.15.27:0/2513109965 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f631415cb10 msgr2=0x7f63140a60c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:09.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.525+0000 7f6313fff700 1 --2- 172.21.15.27:0/2513109965 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f631415cb10 0x7f63140a60c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:09.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.525+0000 7f6313fff700 1 -- 172.21.15.27:0/2513109965 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63140b4000 msgr2=0x7f63140a5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:09.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.525+0000 7f6313fff700 1 --2- 172.21.15.27:0/2513109965 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63140b4000 0x7f63140a5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:09.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.525+0000 7f6313fff700 1 -- 172.21.15.27:0/2513109965 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f630c009a20 con 0x7f63140b1690 2023-10-06T01:41:09.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.526+0000 7f6313fff700 1 --2- 172.21.15.27:0/2513109965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63140b1690 0x7f6314157460 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f631415a3a0 tx=0x7f6308017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:09.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.526+0000 7f62fbfff700 1 -- 172.21.15.27:0/2513109965 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630800bbe0 con 0x7f63140b1690 2023-10-06T01:41:09.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.526+0000 7f62fbfff700 1 -- 172.21.15.27:0/2513109965 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f630801a450 con 0x7f63140b1690 2023-10-06T01:41:09.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.526+0000 7f631a9d2700 1 -- 172.21.15.27:0/2513109965 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63080094f0 con 0x7f63140b1690 2023-10-06T01:41:09.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.526+0000 7f62fbfff700 1 -- 172.21.15.27:0/2513109965 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f630801a7f0 con 0x7f63140b1690 2023-10-06T01:41:09.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.526+0000 7f631a9d2700 1 -- 172.21.15.27:0/2513109965 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63140cd920 con 0x7f63140b1690 2023-10-06T01:41:09.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.527+0000 7f631a9d2700 1 -- 172.21.15.27:0/2513109965 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6314064050 con 0x7f63140b1690 2023-10-06T01:41:09.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.527+0000 7f62fbfff700 1 -- 172.21.15.27:0/2513109965 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f630801ab60 con 0x7f63140b1690 2023-10-06T01:41:09.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.528+0000 7f62fbfff700 1 --2- 172.21.15.27:0/2513109965 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6300066f30 0x7f63000693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:09.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.528+0000 7f62fbfff700 1 -- 172.21.15.27:0/2513109965 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f6308085d90 con 0x7f63140b1690 2023-10-06T01:41:09.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.528+0000 7f63137fe700 1 --2- 172.21.15.27:0/2513109965 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6300066f30 0x7f63000693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:09.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.528+0000 7f63137fe700 1 --2- 172.21.15.27:0/2513109965 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6300066f30 0x7f63000693e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6304007fd0 tx=0x7f6304007d70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:09.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.532+0000 7f62fbfff700 1 -- 172.21.15.27:0/2513109965 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f63080531f0 con 0x7f63140b1690 2023-10-06T01:41:09.725 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.724+0000 7f631a9d2700 1 -- 172.21.15.27:0/2513109965 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f63140b0020 con 0x7f63140b1690 2023-10-06T01:41:09.725 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.725+0000 7f62fbfff700 1 -- 172.21.15.27:0/2513109965 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6308056810 con 0x7f63140b1690 2023-10-06T01:41:09.725 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2023-10-06T01:41:09.727 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.727+0000 7f631a9d2700 1 -- 172.21.15.27:0/2513109965 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6300066f30 msgr2=0x7f63000693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:09.728 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.727+0000 7f631a9d2700 1 --2- 172.21.15.27:0/2513109965 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6300066f30 0x7f63000693e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f6304007fd0 tx=0x7f6304007d70 comp rx=0 tx=0).stop 2023-10-06T01:41:09.728 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.727+0000 7f631a9d2700 1 -- 172.21.15.27:0/2513109965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63140b1690 msgr2=0x7f6314157460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:09.728 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.727+0000 7f631a9d2700 1 --2- 172.21.15.27:0/2513109965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63140b1690 0x7f6314157460 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f631415a3a0 tx=0x7f6308017000 comp rx=0 tx=0).stop 2023-10-06T01:41:09.728 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.727+0000 7f631a9d2700 1 -- 172.21.15.27:0/2513109965 shutdown_connections 2023-10-06T01:41:09.728 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.727+0000 7f631a9d2700 1 --2- 172.21.15.27:0/2513109965 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6300066f30 0x7f63000693e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:09.728 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.727+0000 7f631a9d2700 1 --2- 172.21.15.27:0/2513109965 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f631415cb10 0x7f63140a60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:09.728 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.727+0000 7f631a9d2700 1 --2- 172.21.15.27:0/2513109965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f63140b1690 0x7f6314157460 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:09.728 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.727+0000 7f631a9d2700 1 --2- 172.21.15.27:0/2513109965 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f63140b4000 0x7f63140a5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:09.728 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.727+0000 7f631a9d2700 1 -- 172.21.15.27:0/2513109965 >> 172.21.15.27:0/2513109965 conn(0x7f631406d890 msgr2=0x7f631406dc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:09.728 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.727+0000 7f631a9d2700 1 -- 172.21.15.27:0/2513109965 shutdown_connections 2023-10-06T01:41:09.728 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.727+0000 7f631a9d2700 1 -- 172.21.15.27:0/2513109965 wait complete. 2023-10-06T01:41:09.739 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-06T01:41:09.858 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.858+0000 7f42eae61700 1 Processor -- start 2023-10-06T01:41:09.858 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.858+0000 7f42eae61700 1 -- start start 2023-10-06T01:41:09.860 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.859+0000 7f42eae61700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f42e40c79b0 0x7f42e406c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:09.860 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.859+0000 7f42eae61700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f42e406c6d0 0x7f42e40aed50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:09.861 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.859+0000 7f42eae61700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42e40cce40 con 0x7f42e41571d0 2023-10-06T01:41:09.861 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.859+0000 7f42eae61700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42e406cb50 con 0x7f42e406c6d0 2023-10-06T01:41:09.861 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.859+0000 7f42eae61700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42e40cbf50 con 0x7f42e40c79b0 2023-10-06T01:41:09.861 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.859+0000 7f42e93fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f42e406c6d0 0x7f42e40aed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:09.861 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.859+0000 7f42e8bfd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f42e41571d0 0x7f42e415a740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47308/0 (socket says 172.21.15.27:47308) 2023-10-06T01:41:09.861 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.859+0000 7f42e8bfd700 1 -- 172.21.15.27:0/779786021 learned_addr learned my addr 172.21.15.27:0/779786021 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:09.861 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.859+0000 7f42e3fff700 1 --2- 172.21.15.27:0/779786021 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f42e40c79b0 0x7f42e406c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:09.861 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.859+0000 7f42e93fe700 1 -- 172.21.15.27:0/779786021 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f42e40c79b0 msgr2=0x7f42e406c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:09.861 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.859+0000 7f42e93fe700 1 --2- 172.21.15.27:0/779786021 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f42e40c79b0 0x7f42e406c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:09.861 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.859+0000 7f42e93fe700 1 -- 172.21.15.27:0/779786021 >> v1:172.21.15.27:6789/0 conn(0x7f42e41571d0 legacy=0x7f42e415a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:09.861 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.859+0000 7f42e93fe700 1 -- 172.21.15.27:0/779786021 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42e40cc230 con 0x7f42e406c6d0 2023-10-06T01:41:09.862 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.859+0000 7f42e37fe700 1 -- 172.21.15.27:0/779786021 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950275201 0 0) 0x7f42e40cce40 con 0x7f42e41571d0 2023-10-06T01:41:09.862 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.859+0000 7f42e93fe700 1 --2- 172.21.15.27:0/779786021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42e406c6d0 0x7f42e40aed50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f42cc0098e0 tx=0x7f42cc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3da7c4e42c409a18 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:09.862 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.859+0000 7f42e37fe700 1 -- 172.21.15.27:0/779786021 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42cc009e80 con 0x7f42e406c6d0 2023-10-06T01:41:09.862 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.859+0000 7f42e37fe700 1 -- 172.21.15.27:0/779786021 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42cc00b860 con 0x7f42e406c6d0 2023-10-06T01:41:09.862 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.859+0000 7f42e37fe700 1 -- 172.21.15.27:0/779786021 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42cc00bbe0 con 0x7f42e406c6d0 2023-10-06T01:41:09.862 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.860+0000 7f42eae61700 1 -- 172.21.15.27:0/779786021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42e406c6d0 msgr2=0x7f42e40aed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:09.862 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.860+0000 7f42eae61700 1 --2- 172.21.15.27:0/779786021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42e406c6d0 0x7f42e40aed50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f42cc0098e0 tx=0x7f42cc0091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:09.862 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.860+0000 7f42eae61700 1 -- 172.21.15.27:0/779786021 shutdown_connections 2023-10-06T01:41:09.862 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.860+0000 7f42eae61700 1 --2- 172.21.15.27:0/779786021 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42e406c6d0 0x7f42e40aed50 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:09.862 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.860+0000 7f42eae61700 1 --2- 172.21.15.27:0/779786021 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f42e40c79b0 0x7f42e406c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:09.862 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.860+0000 7f42eae61700 1 -- 172.21.15.27:0/779786021 >> 172.21.15.27:0/779786021 conn(0x7f42e406e6a0 msgr2=0x7f42e406eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:09.862 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.860+0000 7f42eae61700 1 -- 172.21.15.27:0/779786021 shutdown_connections 2023-10-06T01:41:09.863 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.860+0000 7f42eae61700 1 -- 172.21.15.27:0/779786021 wait complete. 2023-10-06T01:41:09.865 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.861+0000 7f42eae61700 1 Processor -- start 2023-10-06T01:41:09.865 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.861+0000 7f42eae61700 1 -- start start 2023-10-06T01:41:09.865 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.862+0000 7f42eae61700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42e406c6d0 0x7f42e414ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:09.865 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.862+0000 7f42eae61700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f42e40c79b0 0x7f42e414e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:09.865 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.862+0000 7f42eae61700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f42e41571d0 0x7f42e4151e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:09.865 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.862+0000 7f42eae61700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e41545a0 con 0x7f42e406c6d0 2023-10-06T01:41:09.866 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.862+0000 7f42eae61700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e40c6c30 con 0x7f42e40c79b0 2023-10-06T01:41:09.866 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.862+0000 7f42eae61700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42e40c6f10 con 0x7f42e41571d0 2023-10-06T01:41:09.866 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.862+0000 7f42e8bfd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42e406c6d0 0x7f42e414ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:09.866 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.862+0000 7f42e8bfd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42e406c6d0 0x7f42e414ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40304/0 (socket says 172.21.15.27:40304) 2023-10-06T01:41:09.866 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.862+0000 7f42e8bfd700 1 -- 172.21.15.27:0/2281515489 learned_addr learned my addr 172.21.15.27:0/2281515489 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:09.866 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.862+0000 7f42e93fe700 1 --2- 172.21.15.27:0/2281515489 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f42e41571d0 0x7f42e4151e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:09.866 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.862+0000 7f42e3fff700 1 --2- 172.21.15.27:0/2281515489 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f42e40c79b0 0x7f42e414e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:09.866 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.862+0000 7f42e8bfd700 1 -- 172.21.15.27:0/2281515489 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f42e41571d0 msgr2=0x7f42e4151e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:09.866 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.862+0000 7f42e8bfd700 1 --2- 172.21.15.27:0/2281515489 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f42e41571d0 0x7f42e4151e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:09.866 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.862+0000 7f42e8bfd700 1 -- 172.21.15.27:0/2281515489 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f42e40c79b0 msgr2=0x7f42e414e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:09.866 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.862+0000 7f42e8bfd700 1 --2- 172.21.15.27:0/2281515489 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f42e40c79b0 0x7f42e414e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:09.866 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.862+0000 7f42e8bfd700 1 -- 172.21.15.27:0/2281515489 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42cc009420 con 0x7f42e406c6d0 2023-10-06T01:41:09.867 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.863+0000 7f42e8bfd700 1 --2- 172.21.15.27:0/2281515489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42e406c6d0 0x7f42e414ec40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f42dc00cc70 tx=0x7f42dc001a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:09.867 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.863+0000 7f42e0ff9700 1 -- 172.21.15.27:0/2281515489 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42dc00ce80 con 0x7f42e406c6d0 2023-10-06T01:41:09.867 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.863+0000 7f42e0ff9700 1 -- 172.21.15.27:0/2281515489 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42dc00ab70 con 0x7f42e406c6d0 2023-10-06T01:41:09.867 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.863+0000 7f42eae61700 1 -- 172.21.15.27:0/2281515489 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42e4153f50 con 0x7f42e406c6d0 2023-10-06T01:41:09.867 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.863+0000 7f42e0ff9700 1 -- 172.21.15.27:0/2281515489 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42dc0070f0 con 0x7f42e406c6d0 2023-10-06T01:41:09.867 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.863+0000 7f42eae61700 1 -- 172.21.15.27:0/2281515489 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42e414f3c0 con 0x7f42e406c6d0 2023-10-06T01:41:09.867 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.864+0000 7f42eae61700 1 -- 172.21.15.27:0/2281515489 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42e4064050 con 0x7f42e406c6d0 2023-10-06T01:41:09.867 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.864+0000 7f42e0ff9700 1 -- 172.21.15.27:0/2281515489 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f42dc0074e0 con 0x7f42e406c6d0 2023-10-06T01:41:09.867 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.865+0000 7f42e0ff9700 1 --2- 172.21.15.27:0/2281515489 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f42c4066f30 0x7f42c40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:09.867 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.865+0000 7f42e0ff9700 1 -- 172.21.15.27:0/2281515489 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f42dc0847c0 con 0x7f42e406c6d0 2023-10-06T01:41:09.867 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.865+0000 7f42e3fff700 1 --2- 172.21.15.27:0/2281515489 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f42c4066f30 0x7f42c40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:09.868 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.865+0000 7f42e3fff700 1 --2- 172.21.15.27:0/2281515489 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f42c4066f30 0x7f42c40693e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f42e40b1d00 tx=0x7f42d4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:09.869 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:09.869+0000 7f42e0ff9700 1 -- 172.21.15.27:0/2281515489 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f42dc051ca0 con 0x7f42e406c6d0 2023-10-06T01:41:10.064 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.064+0000 7f42eae61700 1 -- 172.21.15.27:0/2281515489 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f42e40cc510 con 0x7f42e406c6d0 2023-10-06T01:41:10.064 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.064+0000 7f42e0ff9700 1 -- 172.21.15.27:0/2281515489 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+126 (secure 0 0 0) 0x7f42dc0552c0 con 0x7f42e406c6d0 2023-10-06T01:41:10.064 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:10.064 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2023-10-06T01:41:10.065 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:41:10.065 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2023-10-06T01:41:10.065 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": "rbd" 2023-10-06T01:41:10.065 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2023-10-06T01:41:10.065 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:41:10.065 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 2, 2023-10-06T01:41:10.065 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2023-10-06T01:41:10.065 INFO:teuthology.orchestra.run.smithi027.stdout: } 2023-10-06T01:41:10.065 INFO:teuthology.orchestra.run.smithi027.stdout:] 2023-10-06T01:41:10.066 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.066+0000 7f42eae61700 1 -- 172.21.15.27:0/2281515489 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f42c4066f30 msgr2=0x7f42c40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:10.066 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.066+0000 7f42eae61700 1 --2- 172.21.15.27:0/2281515489 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f42c4066f30 0x7f42c40693e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f42e40b1d00 tx=0x7f42d4008040 comp rx=0 tx=0).stop 2023-10-06T01:41:10.066 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.066+0000 7f42eae61700 1 -- 172.21.15.27:0/2281515489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42e406c6d0 msgr2=0x7f42e414ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:10.067 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.066+0000 7f42eae61700 1 --2- 172.21.15.27:0/2281515489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42e406c6d0 0x7f42e414ec40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f42dc00cc70 tx=0x7f42dc001a30 comp rx=0 tx=0).stop 2023-10-06T01:41:10.067 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.066+0000 7f42eae61700 1 -- 172.21.15.27:0/2281515489 shutdown_connections 2023-10-06T01:41:10.067 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.066+0000 7f42eae61700 1 --2- 172.21.15.27:0/2281515489 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f42c4066f30 0x7f42c40693e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:10.067 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.066+0000 7f42eae61700 1 --2- 172.21.15.27:0/2281515489 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f42e41571d0 0x7f42e4151e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:10.067 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.066+0000 7f42eae61700 1 --2- 172.21.15.27:0/2281515489 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f42e406c6d0 0x7f42e414ec40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:10.067 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.066+0000 7f42eae61700 1 --2- 172.21.15.27:0/2281515489 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f42e40c79b0 0x7f42e414e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:10.067 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.066+0000 7f42eae61700 1 -- 172.21.15.27:0/2281515489 >> 172.21.15.27:0/2281515489 conn(0x7f42e406e6a0 msgr2=0x7f42e40c2630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:10.067 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.066+0000 7f42eae61700 1 -- 172.21.15.27:0/2281515489 shutdown_connections 2023-10-06T01:41:10.067 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.066+0000 7f42eae61700 1 -- 172.21.15.27:0/2281515489 wait complete. 2023-10-06T01:41:10.078 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-06T01:41:10.079 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-06T01:41:10.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.201+0000 7f7297b22700 1 Processor -- start 2023-10-06T01:41:10.202 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.201+0000 7f7297b22700 1 -- start start 2023-10-06T01:41:10.204 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.202+0000 7f7297b22700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f72900cb6b0 0x7f7290159d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:10.204 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.202+0000 7f7297b22700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f72900695e0 0x7f72900aa270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:10.204 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.202+0000 7f7297b22700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72900a9de0 con 0x7f729015a240 2023-10-06T01:41:10.204 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.202+0000 7f7297b22700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f729015a7b0 con 0x7f72900cb6b0 2023-10-06T01:41:10.204 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.202+0000 7f7297b22700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f729015aa90 con 0x7f72900695e0 2023-10-06T01:41:10.204 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.202+0000 7f72958be700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f72900cb6b0 0x7f7290159d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:10.204 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.202+0000 7f72958be700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f72900cb6b0 0x7f7290159d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40306/0 (socket says 172.21.15.27:40306) 2023-10-06T01:41:10.204 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.202+0000 7f72950bd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f729015a240 0x7f7290068ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47314/0 (socket says 172.21.15.27:47314) 2023-10-06T01:41:10.204 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.202+0000 7f72958be700 1 -- 172.21.15.27:0/1407123522 learned_addr learned my addr 172.21.15.27:0/1407123522 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:10.205 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.202+0000 7f72960bf700 1 --2- 172.21.15.27:0/1407123522 >> v2:172.21.15.124:3300/0 conn(0x7f72900695e0 0x7f72900aa270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:10.205 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.202+0000 7f72958be700 1 -- 172.21.15.27:0/1407123522 >> v2:172.21.15.124:3300/0 conn(0x7f72900695e0 msgr2=0x7f72900aa270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:10.205 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.202+0000 7f72958be700 1 --2- 172.21.15.27:0/1407123522 >> v2:172.21.15.124:3300/0 conn(0x7f72900695e0 0x7f72900aa270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:10.205 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.202+0000 7f72958be700 1 -- 172.21.15.27:0/1407123522 >> v1:172.21.15.27:6789/0 conn(0x7f729015a240 legacy=0x7f7290068ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:10.205 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.202+0000 7f72948bc700 1 -- 172.21.15.27:0/1407123522 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4069610530 0 0) 0x7f72900a9de0 con 0x7f729015a240 2023-10-06T01:41:10.205 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.202+0000 7f72958be700 1 -- 172.21.15.27:0/1407123522 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72900a9a70 con 0x7f72900cb6b0 2023-10-06T01:41:10.205 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.203+0000 7f72958be700 1 --2- 172.21.15.27:0/1407123522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72900cb6b0 0x7f7290159d00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f72900ccae0 tx=0x7f7288009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1520e226ef3bab99 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:10.205 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.203+0000 7f72948bc700 1 -- 172.21.15.27:0/1407123522 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7288009e70 con 0x7f72900cb6b0 2023-10-06T01:41:10.205 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.203+0000 7f72948bc700 1 -- 172.21.15.27:0/1407123522 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f728800b860 con 0x7f72900cb6b0 2023-10-06T01:41:10.205 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.203+0000 7f72948bc700 1 -- 172.21.15.27:0/1407123522 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f728800bc00 con 0x7f72900cb6b0 2023-10-06T01:41:10.206 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.203+0000 7f7297b22700 1 -- 172.21.15.27:0/1407123522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72900cb6b0 msgr2=0x7f7290159d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:10.206 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.203+0000 7f7297b22700 1 --2- 172.21.15.27:0/1407123522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72900cb6b0 0x7f7290159d00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f72900ccae0 tx=0x7f7288009250 comp rx=0 tx=0).stop 2023-10-06T01:41:10.206 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.203+0000 7f7297b22700 1 -- 172.21.15.27:0/1407123522 shutdown_connections 2023-10-06T01:41:10.206 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.203+0000 7f7297b22700 1 --2- 172.21.15.27:0/1407123522 >> v2:172.21.15.124:3300/0 conn(0x7f72900695e0 0x7f72900aa270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:10.206 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.203+0000 7f7297b22700 1 --2- 172.21.15.27:0/1407123522 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f72900cb6b0 0x7f7290159d00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:10.206 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.203+0000 7f7297b22700 1 -- 172.21.15.27:0/1407123522 >> 172.21.15.27:0/1407123522 conn(0x7f72900ca070 msgr2=0x7f729006df00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:10.206 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.203+0000 7f7297b22700 1 -- 172.21.15.27:0/1407123522 shutdown_connections 2023-10-06T01:41:10.206 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.203+0000 7f7297b22700 1 -- 172.21.15.27:0/1407123522 wait complete. 2023-10-06T01:41:10.209 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.204+0000 7f7297b22700 1 Processor -- start 2023-10-06T01:41:10.209 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.204+0000 7f7297b22700 1 -- start start 2023-10-06T01:41:10.209 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.205+0000 7f7297b22700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f72900695e0 0x7f72900c6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:10.209 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.205+0000 7f7297b22700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f72900cb6b0 0x7f7290151290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:10.209 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.205+0000 7f7297b22700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f729015a240 0x7f72901517d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:10.209 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.205+0000 7f7297b22700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7290151d10 con 0x7f729015a240 2023-10-06T01:41:10.209 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.205+0000 7f7297b22700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72900b1820 con 0x7f72900695e0 2023-10-06T01:41:10.209 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.205+0000 7f7297b22700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72900b1b00 con 0x7f72900cb6b0 2023-10-06T01:41:10.210 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.205+0000 7f72960bf700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f729015a240 0x7f72901517d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:10.210 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.205+0000 7f72960bf700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f729015a240 0x7f72901517d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40314/0 (socket says 172.21.15.27:40314) 2023-10-06T01:41:10.210 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.205+0000 7f72960bf700 1 -- 172.21.15.27:0/973700461 learned_addr learned my addr 172.21.15.27:0/973700461 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:10.210 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.205+0000 7f72958be700 1 --2- 172.21.15.27:0/973700461 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f72900695e0 0x7f72900c6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:10.210 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.205+0000 7f72950bd700 1 --2- 172.21.15.27:0/973700461 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f72900cb6b0 0x7f7290151290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:10.210 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.206+0000 7f72960bf700 1 -- 172.21.15.27:0/973700461 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f72900cb6b0 msgr2=0x7f7290151290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:10.210 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.206+0000 7f72960bf700 1 --2- 172.21.15.27:0/973700461 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f72900cb6b0 0x7f7290151290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:10.210 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.206+0000 7f72960bf700 1 -- 172.21.15.27:0/973700461 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f72900695e0 msgr2=0x7f72900c6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:10.210 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.206+0000 7f72960bf700 1 --2- 172.21.15.27:0/973700461 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f72900695e0 0x7f72900c6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:10.210 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.206+0000 7f72960bf700 1 -- 172.21.15.27:0/973700461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72880094f0 con 0x7f729015a240 2023-10-06T01:41:10.210 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.206+0000 7f72960bf700 1 --2- 172.21.15.27:0/973700461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f729015a240 0x7f72901517d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7280009d70 tx=0x7f728000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:10.210 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.206+0000 7f728dffb700 1 -- 172.21.15.27:0/973700461 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f728000caa0 con 0x7f729015a240 2023-10-06T01:41:10.211 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.206+0000 7f728dffb700 1 -- 172.21.15.27:0/973700461 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f728000f040 con 0x7f729015a240 2023-10-06T01:41:10.211 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.206+0000 7f7297b22700 1 -- 172.21.15.27:0/973700461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72900b1de0 con 0x7f729015a240 2023-10-06T01:41:10.211 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.206+0000 7f728dffb700 1 -- 172.21.15.27:0/973700461 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f728000ce50 con 0x7f729015a240 2023-10-06T01:41:10.211 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.206+0000 7f7297b22700 1 -- 172.21.15.27:0/973700461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72900c7310 con 0x7f729015a240 2023-10-06T01:41:10.211 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.207+0000 7f7297b22700 1 -- 172.21.15.27:0/973700461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7290151e80 con 0x7f729015a240 2023-10-06T01:41:10.211 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.207+0000 7f728dffb700 1 -- 172.21.15.27:0/973700461 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7280014690 con 0x7f729015a240 2023-10-06T01:41:10.211 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.208+0000 7f728dffb700 1 --2- 172.21.15.27:0/973700461 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f727c066f30 0x7f727c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:10.211 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.208+0000 7f728dffb700 1 -- 172.21.15.27:0/973700461 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f7280084e60 con 0x7f729015a240 2023-10-06T01:41:10.211 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.208+0000 7f72958be700 1 --2- 172.21.15.27:0/973700461 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f727c066f30 0x7f727c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:10.211 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.208+0000 7f72958be700 1 --2- 172.21.15.27:0/973700461 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f727c066f30 0x7f727c0693e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f72900ccae0 tx=0x7f7288017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:10.212 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.212+0000 7f728dffb700 1 -- 172.21.15.27:0/973700461 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f72800522c0 con 0x7f729015a240 2023-10-06T01:41:10.404 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:10.404+0000 7f7297b22700 1 -- 172.21.15.27:0/973700461 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f729016ae20 con 0x7f729015a240 2023-10-06T01:41:11.317 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.317+0000 7f728dffb700 1 -- 172.21.15.27:0/973700461 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f729016ae20 con 0x7f729015a240 2023-10-06T01:41:11.322 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.322+0000 7f7297b22700 1 -- 172.21.15.27:0/973700461 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f727c066f30 msgr2=0x7f727c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:11.322 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.322+0000 7f7297b22700 1 --2- 172.21.15.27:0/973700461 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f727c066f30 0x7f727c0693e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f72900ccae0 tx=0x7f7288017000 comp rx=0 tx=0).stop 2023-10-06T01:41:11.322 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.322+0000 7f7297b22700 1 -- 172.21.15.27:0/973700461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f729015a240 msgr2=0x7f72901517d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:11.323 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.322+0000 7f7297b22700 1 --2- 172.21.15.27:0/973700461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f729015a240 0x7f72901517d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7280009d70 tx=0x7f728000b040 comp rx=0 tx=0).stop 2023-10-06T01:41:11.323 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.323+0000 7f7297b22700 1 -- 172.21.15.27:0/973700461 shutdown_connections 2023-10-06T01:41:11.323 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.323+0000 7f7297b22700 1 --2- 172.21.15.27:0/973700461 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f727c066f30 0x7f727c0693e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:11.323 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.323+0000 7f7297b22700 1 --2- 172.21.15.27:0/973700461 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f729015a240 0x7f72901517d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:11.323 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.323+0000 7f7297b22700 1 --2- 172.21.15.27:0/973700461 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f72900695e0 0x7f72900c6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:11.324 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.323+0000 7f7297b22700 1 --2- 172.21.15.27:0/973700461 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f72900cb6b0 0x7f7290151290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:11.324 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.323+0000 7f7297b22700 1 -- 172.21.15.27:0/973700461 >> 172.21.15.27:0/973700461 conn(0x7f72900ca070 msgr2=0x7f7290152710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:11.324 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.323+0000 7f7297b22700 1 -- 172.21.15.27:0/973700461 shutdown_connections 2023-10-06T01:41:11.324 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.323+0000 7f7297b22700 1 -- 172.21.15.27:0/973700461 wait complete. 2023-10-06T01:41:11.324 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2023-10-06T01:41:11.337 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-06T01:41:11.466 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.466+0000 7facc21a5700 1 Processor -- start 2023-10-06T01:41:11.466 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.466+0000 7facc21a5700 1 -- start start 2023-10-06T01:41:11.469 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.467+0000 7facc21a5700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7facbc159890 0x7facbc0b0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:11.469 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.467+0000 7facc21a5700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7facbc0cdcf0 0x7facbc0b14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:11.469 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.467+0000 7facc21a5700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7facbc0b1a10 0x7facbc0d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:11.469 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.467+0000 7facc21a5700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facbc0ce190 con 0x7facbc159890 2023-10-06T01:41:11.469 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.467+0000 7facc21a5700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facbc0b1e70 con 0x7facbc0b1a10 2023-10-06T01:41:11.469 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.467+0000 7facc21a5700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facbc0cd860 con 0x7facbc0cdcf0 2023-10-06T01:41:11.469 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.467+0000 7facbb7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7facbc159890 0x7facbc0b0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:11.469 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.467+0000 7facbb7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7facbc159890 0x7facbc0b0eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40324/0 (socket says 172.21.15.27:40324) 2023-10-06T01:41:11.469 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.467+0000 7facbb7fe700 1 -- 172.21.15.27:0/2240176533 learned_addr learned my addr 172.21.15.27:0/2240176533 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:11.469 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.467+0000 7facbbfff700 1 --2- 172.21.15.27:0/2240176533 >> v2:172.21.15.124:3300/0 conn(0x7facbc0b1a10 0x7facbc0d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:11.469 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.467+0000 7facbaffd700 1 --2- 172.21.15.27:0/2240176533 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7facbc0cdcf0 0x7facbc0b14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.467+0000 7facbb7fe700 1 -- 172.21.15.27:0/2240176533 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7facbc0cdcf0 msgr2=0x7facbc0b14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.467+0000 7facbb7fe700 1 --2- 172.21.15.27:0/2240176533 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7facbc0cdcf0 0x7facbc0b14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.467+0000 7facbb7fe700 1 -- 172.21.15.27:0/2240176533 >> v2:172.21.15.124:3300/0 conn(0x7facbc0b1a10 msgr2=0x7facbc0d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.467+0000 7facbb7fe700 1 --2- 172.21.15.27:0/2240176533 >> v2:172.21.15.124:3300/0 conn(0x7facbc0b1a10 0x7facbc0d0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.467+0000 7facbb7fe700 1 -- 172.21.15.27:0/2240176533 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facbc0ccc50 con 0x7facbc159890 2023-10-06T01:41:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.467+0000 7facbaffd700 1 --2- 172.21.15.27:0/2240176533 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7facbc0cdcf0 0x7facbc0b14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.467+0000 7facbbfff700 1 --2- 172.21.15.27:0/2240176533 >> v2:172.21.15.124:3300/0 conn(0x7facbc0b1a10 0x7facbc0d0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.467+0000 7facbb7fe700 1 --2- 172.21.15.27:0/2240176533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facbc159890 0x7facbc0b0eb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7facbc089ac0 tx=0x7facac009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2a1a7735b6e4ce7 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.467+0000 7facba7fc700 1 -- 172.21.15.27:0/2240176533 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facac005c10 con 0x7facbc159890 2023-10-06T01:41:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.468+0000 7facba7fc700 1 -- 172.21.15.27:0/2240176533 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facac005d70 con 0x7facbc159890 2023-10-06T01:41:11.470 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.468+0000 7facba7fc700 1 -- 172.21.15.27:0/2240176533 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facac00b880 con 0x7facbc159890 2023-10-06T01:41:11.471 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.468+0000 7facc21a5700 1 -- 172.21.15.27:0/2240176533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facbc159890 msgr2=0x7facbc0b0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:11.471 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.468+0000 7facc21a5700 1 --2- 172.21.15.27:0/2240176533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facbc159890 0x7facbc0b0eb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7facbc089ac0 tx=0x7facac009220 comp rx=0 tx=0).stop 2023-10-06T01:41:11.471 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.468+0000 7facc21a5700 1 -- 172.21.15.27:0/2240176533 shutdown_connections 2023-10-06T01:41:11.471 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.468+0000 7facc21a5700 1 --2- 172.21.15.27:0/2240176533 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facbc159890 0x7facbc0b0eb0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:11.471 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.468+0000 7facc21a5700 1 --2- 172.21.15.27:0/2240176533 >> v2:172.21.15.124:3300/0 conn(0x7facbc0b1a10 0x7facbc0d0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:11.471 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.468+0000 7facc21a5700 1 --2- 172.21.15.27:0/2240176533 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7facbc0cdcf0 0x7facbc0b14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:11.471 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.468+0000 7facc21a5700 1 -- 172.21.15.27:0/2240176533 >> 172.21.15.27:0/2240176533 conn(0x7facbc099990 msgr2=0x7facbc0b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:11.471 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.468+0000 7facc21a5700 1 -- 172.21.15.27:0/2240176533 shutdown_connections 2023-10-06T01:41:11.471 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.468+0000 7facc21a5700 1 -- 172.21.15.27:0/2240176533 wait complete. 2023-10-06T01:41:11.474 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.469+0000 7facc21a5700 1 Processor -- start 2023-10-06T01:41:11.474 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.469+0000 7facc21a5700 1 -- start start 2023-10-06T01:41:11.474 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.470+0000 7facc21a5700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7facbc0b1a10 0x7facbc14fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:11.474 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.470+0000 7facc21a5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facbc0cdcf0 0x7facbc067ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:11.474 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.470+0000 7facc21a5700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7facbc159890 0x7facbc068010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:11.474 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.470+0000 7facc21a5700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facbc151760 con 0x7facbc0cdcf0 2023-10-06T01:41:11.474 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.470+0000 7facc21a5700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facbc151170 con 0x7facbc0b1a10 2023-10-06T01:41:11.474 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.470+0000 7facc21a5700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facbc15f1e0 con 0x7facbc159890 2023-10-06T01:41:11.474 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.470+0000 7facbaffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facbc0cdcf0 0x7facbc067ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:11.475 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.470+0000 7facbaffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facbc0cdcf0 0x7facbc067ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40328/0 (socket says 172.21.15.27:40328) 2023-10-06T01:41:11.475 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.470+0000 7facbaffd700 1 -- 172.21.15.27:0/2021945038 learned_addr learned my addr 172.21.15.27:0/2021945038 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:11.475 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.470+0000 7facbb7fe700 1 --2- 172.21.15.27:0/2021945038 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7facbc0b1a10 0x7facbc14fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:11.475 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.470+0000 7facbbfff700 1 --2- 172.21.15.27:0/2021945038 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7facbc159890 0x7facbc068010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:11.475 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.470+0000 7facbaffd700 1 -- 172.21.15.27:0/2021945038 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7facbc159890 msgr2=0x7facbc068010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:11.475 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.470+0000 7facbaffd700 1 --2- 172.21.15.27:0/2021945038 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7facbc159890 0x7facbc068010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:11.475 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.470+0000 7facbaffd700 1 -- 172.21.15.27:0/2021945038 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7facbc0b1a10 msgr2=0x7facbc14fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:11.475 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.470+0000 7facbaffd700 1 --2- 172.21.15.27:0/2021945038 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7facbc0b1a10 0x7facbc14fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:11.475 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.470+0000 7facbaffd700 1 -- 172.21.15.27:0/2021945038 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facac009480 con 0x7facbc0cdcf0 2023-10-06T01:41:11.476 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.470+0000 7facbbfff700 1 --2- 172.21.15.27:0/2021945038 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7facbc159890 0x7facbc068010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:41:11.476 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.470+0000 7facbb7fe700 1 --2- 172.21.15.27:0/2021945038 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7facbc0b1a10 0x7facbc14fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:41:11.476 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.471+0000 7facbaffd700 1 --2- 172.21.15.27:0/2021945038 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facbc0cdcf0 0x7facbc067ad0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7facb000e850 tx=0x7facb0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:11.476 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.471+0000 7faca37fe700 1 -- 172.21.15.27:0/2021945038 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb0013410 con 0x7facbc0cdcf0 2023-10-06T01:41:11.476 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.471+0000 7faca37fe700 1 -- 172.21.15.27:0/2021945038 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facb0013570 con 0x7facbc0cdcf0 2023-10-06T01:41:11.476 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.471+0000 7faca37fe700 1 -- 172.21.15.27:0/2021945038 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facb0013910 con 0x7facbc0cdcf0 2023-10-06T01:41:11.476 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.471+0000 7facc21a5700 1 -- 172.21.15.27:0/2021945038 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facbc15f4c0 con 0x7facbc0cdcf0 2023-10-06T01:41:11.476 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.471+0000 7facc21a5700 1 -- 172.21.15.27:0/2021945038 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facbc150060 con 0x7facbc0cdcf0 2023-10-06T01:41:11.476 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.472+0000 7facc21a5700 1 -- 172.21.15.27:0/2021945038 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facbc064050 con 0x7facbc0cdcf0 2023-10-06T01:41:11.476 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.472+0000 7faca37fe700 1 -- 172.21.15.27:0/2021945038 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7facb0013d40 con 0x7facbc0cdcf0 2023-10-06T01:41:11.476 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.473+0000 7faca37fe700 1 --2- 172.21.15.27:0/2021945038 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7faca8066f30 0x7faca80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:11.476 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.473+0000 7faca37fe700 1 -- 172.21.15.27:0/2021945038 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5576+0+0 (secure 0 0 0) 0x7facb00844f0 con 0x7facbc0cdcf0 2023-10-06T01:41:11.476 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.473+0000 7facbb7fe700 1 --2- 172.21.15.27:0/2021945038 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7faca8066f30 0x7faca80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:11.477 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.473+0000 7facbb7fe700 1 --2- 172.21.15.27:0/2021945038 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7faca8066f30 0x7faca80693e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7facac009260 tx=0x7facac016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:11.477 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.477+0000 7faca37fe700 1 -- 172.21.15.27:0/2021945038 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7facb00517e0 con 0x7facbc0cdcf0 2023-10-06T01:41:11.670 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:11.670+0000 7facc21a5700 1 -- 172.21.15.27:0/2021945038 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7facbc1518a0 con 0x7facbc0cdcf0 2023-10-06T01:41:12.329 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.328+0000 7faca37fe700 1 -- 172.21.15.27:0/2021945038 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v18) v1 ==== 170+0+0 (secure 0 0 0) 0x7facb0054e00 con 0x7facbc0cdcf0 2023-10-06T01:41:12.339 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.338+0000 7facc21a5700 1 -- 172.21.15.27:0/2021945038 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7faca8066f30 msgr2=0x7faca80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:12.339 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.338+0000 7facc21a5700 1 --2- 172.21.15.27:0/2021945038 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7faca8066f30 0x7faca80693e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7facac009260 tx=0x7facac016000 comp rx=0 tx=0).stop 2023-10-06T01:41:12.339 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.338+0000 7facc21a5700 1 -- 172.21.15.27:0/2021945038 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facbc0cdcf0 msgr2=0x7facbc067ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:12.339 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.338+0000 7facc21a5700 1 --2- 172.21.15.27:0/2021945038 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facbc0cdcf0 0x7facbc067ad0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7facb000e850 tx=0x7facb0008040 comp rx=0 tx=0).stop 2023-10-06T01:41:12.339 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.338+0000 7facc21a5700 1 -- 172.21.15.27:0/2021945038 shutdown_connections 2023-10-06T01:41:12.339 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.338+0000 7facc21a5700 1 --2- 172.21.15.27:0/2021945038 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7faca8066f30 0x7faca80693e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:12.339 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.338+0000 7facc21a5700 1 --2- 172.21.15.27:0/2021945038 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7facbc159890 0x7facbc068010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:12.339 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.338+0000 7facc21a5700 1 --2- 172.21.15.27:0/2021945038 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7facbc0b1a10 0x7facbc14fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:12.339 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.338+0000 7facc21a5700 1 --2- 172.21.15.27:0/2021945038 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7facbc0cdcf0 0x7facbc067ad0 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7facb000e850 tx=0x7facb0008040 comp rx=0 tx=0).stop 2023-10-06T01:41:12.339 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.338+0000 7facc21a5700 1 -- 172.21.15.27:0/2021945038 >> 172.21.15.27:0/2021945038 conn(0x7facbc099990 msgr2=0x7facbc0d0680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:12.344 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.343+0000 7facc21a5700 1 -- 172.21.15.27:0/2021945038 shutdown_connections 2023-10-06T01:41:12.344 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.344+0000 7facc21a5700 1 -- 172.21.15.27:0/2021945038 wait complete. 2023-10-06T01:41:12.346 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2023-10-06T01:41:12.359 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-06T01:41:12.484 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.483+0000 7fe310095700 1 Processor -- start 2023-10-06T01:41:12.484 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.484+0000 7fe310095700 1 -- start start 2023-10-06T01:41:12.486 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.484+0000 7fe310095700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe308159500 0x7fe3081598d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:12.486 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.484+0000 7fe310095700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe30815dab0 0x7fe3080cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:12.487 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.484+0000 7fe310095700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3080b1110 con 0x7fe3080b3fd0 2023-10-06T01:41:12.487 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.484+0000 7fe310095700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3080b13b0 con 0x7fe308159500 2023-10-06T01:41:12.487 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.484+0000 7fe310095700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3080b1690 con 0x7fe30815dab0 2023-10-06T01:41:12.487 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.485+0000 7fe30d630700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe3080b3fd0 0x7fe30815d3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47316/0 (socket says 172.21.15.27:47316) 2023-10-06T01:41:12.487 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.485+0000 7fe30d630700 1 -- 172.21.15.27:0/4156394422 learned_addr learned my addr 172.21.15.27:0/4156394422 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:12.487 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.485+0000 7fe30e632700 1 --2- 172.21.15.27:0/4156394422 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe30815dab0 0x7fe3080cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:12.487 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.485+0000 7fe30de31700 1 --2- 172.21.15.27:0/4156394422 >> v2:172.21.15.124:3300/0 conn(0x7fe308159500 0x7fe3081598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:12.487 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.485+0000 7fe30ce2f700 1 -- 172.21.15.27:0/4156394422 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3571911486 0 0) 0x7fe3080b1110 con 0x7fe3080b3fd0 2023-10-06T01:41:12.487 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.485+0000 7fe30ce2f700 1 -- 172.21.15.27:0/4156394422 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2f0003570 con 0x7fe3080b3fd0 2023-10-06T01:41:12.487 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.485+0000 7fe30de31700 1 -- 172.21.15.27:0/4156394422 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe30815dab0 msgr2=0x7fe3080cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:12.487 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.485+0000 7fe30de31700 1 --2- 172.21.15.27:0/4156394422 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe30815dab0 0x7fe3080cf6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:12.487 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.485+0000 7fe30de31700 1 -- 172.21.15.27:0/4156394422 >> v1:172.21.15.27:6789/0 conn(0x7fe3080b3fd0 legacy=0x7fe30815d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:12.488 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.485+0000 7fe30de31700 1 -- 172.21.15.27:0/4156394422 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3080b1970 con 0x7fe308159500 2023-10-06T01:41:12.488 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.485+0000 7fe30de31700 1 --2- 172.21.15.27:0/4156394422 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe308159500 0x7fe3081598d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe2f80098e0 tx=0x7fe2f80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1ec5b0ec0ed70f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:12.488 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.485+0000 7fe30ce2f700 1 -- 172.21.15.27:0/4156394422 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f8011630 con 0x7fe308159500 2023-10-06T01:41:12.488 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.485+0000 7fe30ce2f700 1 -- 172.21.15.27:0/4156394422 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2f8011790 con 0x7fe308159500 2023-10-06T01:41:12.488 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.485+0000 7fe30ce2f700 1 -- 172.21.15.27:0/4156394422 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2f8011b10 con 0x7fe308159500 2023-10-06T01:41:12.488 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.486+0000 7fe310095700 1 -- 172.21.15.27:0/4156394422 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe308159500 msgr2=0x7fe3081598d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:12.488 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.486+0000 7fe310095700 1 --2- 172.21.15.27:0/4156394422 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe308159500 0x7fe3081598d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe2f80098e0 tx=0x7fe2f80091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:12.488 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.486+0000 7fe310095700 1 -- 172.21.15.27:0/4156394422 shutdown_connections 2023-10-06T01:41:12.488 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.486+0000 7fe310095700 1 --2- 172.21.15.27:0/4156394422 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe30815dab0 0x7fe3080cf6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:12.489 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.486+0000 7fe310095700 1 --2- 172.21.15.27:0/4156394422 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe308159500 0x7fe3081598d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:12.489 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.486+0000 7fe310095700 1 -- 172.21.15.27:0/4156394422 >> 172.21.15.27:0/4156394422 conn(0x7fe308067630 msgr2=0x7fe308069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:12.489 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.486+0000 7fe310095700 1 -- 172.21.15.27:0/4156394422 shutdown_connections 2023-10-06T01:41:12.489 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.486+0000 7fe310095700 1 -- 172.21.15.27:0/4156394422 wait complete. 2023-10-06T01:41:12.492 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.487+0000 7fe310095700 1 Processor -- start 2023-10-06T01:41:12.492 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.487+0000 7fe310095700 1 -- start start 2023-10-06T01:41:12.492 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.488+0000 7fe310095700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe3080b3fd0 0x7fe3081421f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:12.492 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.488+0000 7fe310095700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe308159500 0x7fe30815b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:12.492 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.488+0000 7fe310095700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe30815dab0 0x7fe3080b0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:12.492 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.488+0000 7fe310095700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe308150760 con 0x7fe30815dab0 2023-10-06T01:41:12.492 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.488+0000 7fe310095700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe308142730 con 0x7fe3080b3fd0 2023-10-06T01:41:12.492 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.488+0000 7fe310095700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe30815bc90 con 0x7fe308159500 2023-10-06T01:41:12.492 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.488+0000 7fe30e632700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe30815dab0 0x7fe3080b0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:12.493 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.488+0000 7fe30e632700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe30815dab0 0x7fe3080b0320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40336/0 (socket says 172.21.15.27:40336) 2023-10-06T01:41:12.493 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.488+0000 7fe30e632700 1 -- 172.21.15.27:0/601325630 learned_addr learned my addr 172.21.15.27:0/601325630 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:12.493 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.488+0000 7fe30d630700 1 --2- 172.21.15.27:0/601325630 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe308159500 0x7fe30815b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:12.493 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.488+0000 7fe30de31700 1 --2- 172.21.15.27:0/601325630 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe3080b3fd0 0x7fe3081421f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:12.493 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.488+0000 7fe30e632700 1 -- 172.21.15.27:0/601325630 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe308159500 msgr2=0x7fe30815b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:12.493 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.488+0000 7fe30e632700 1 --2- 172.21.15.27:0/601325630 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe308159500 0x7fe30815b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:12.493 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.488+0000 7fe30e632700 1 -- 172.21.15.27:0/601325630 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe3080b3fd0 msgr2=0x7fe3081421f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:12.493 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.488+0000 7fe30e632700 1 --2- 172.21.15.27:0/601325630 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe3080b3fd0 0x7fe3081421f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:12.493 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.488+0000 7fe30e632700 1 -- 172.21.15.27:0/601325630 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f8009420 con 0x7fe30815dab0 2023-10-06T01:41:12.493 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.488+0000 7fe30de31700 1 --2- 172.21.15.27:0/601325630 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe3080b3fd0 0x7fe3081421f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:12.493 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.489+0000 7fe30e632700 1 --2- 172.21.15.27:0/601325630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe30815dab0 0x7fe3080b0320 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe300009d70 tx=0x7fe30000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:12.493 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.489+0000 7fe305ffb700 1 -- 172.21.15.27:0/601325630 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30000cb50 con 0x7fe30815dab0 2023-10-06T01:41:12.494 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.489+0000 7fe305ffb700 1 -- 172.21.15.27:0/601325630 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe30000f040 con 0x7fe30815dab0 2023-10-06T01:41:12.494 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.489+0000 7fe310095700 1 -- 172.21.15.27:0/601325630 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe308150110 con 0x7fe30815dab0 2023-10-06T01:41:12.494 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.489+0000 7fe305ffb700 1 -- 172.21.15.27:0/601325630 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe300014480 con 0x7fe30815dab0 2023-10-06T01:41:12.494 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.489+0000 7fe310095700 1 -- 172.21.15.27:0/601325630 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3081531f0 con 0x7fe30815dab0 2023-10-06T01:41:12.494 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.490+0000 7fe310095700 1 -- 172.21.15.27:0/601325630 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe308064050 con 0x7fe30815dab0 2023-10-06T01:41:12.494 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.490+0000 7fe305ffb700 1 -- 172.21.15.27:0/601325630 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe300014830 con 0x7fe30815dab0 2023-10-06T01:41:12.494 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.491+0000 7fe305ffb700 1 --2- 172.21.15.27:0/601325630 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe2f4066f30 0x7fe2f40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:12.494 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.491+0000 7fe305ffb700 1 -- 172.21.15.27:0/601325630 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5958+0+0 (secure 0 0 0) 0x7fe3000852d0 con 0x7fe30815dab0 2023-10-06T01:41:12.494 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.491+0000 7fe30de31700 1 --2- 172.21.15.27:0/601325630 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe2f4066f30 0x7fe2f40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:12.494 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.491+0000 7fe30de31700 1 --2- 172.21.15.27:0/601325630 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe2f4066f30 0x7fe2f40693e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe2f80091e0 tx=0x7fe2f8017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:12.495 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.495+0000 7fe305ffb700 1 -- 172.21.15.27:0/601325630 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe300052440 con 0x7fe30815dab0 2023-10-06T01:41:12.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:12.734+0000 7fe310095700 1 -- 172.21.15.27:0/601325630 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fe3080b4470 con 0x7fe30815dab0 2023-10-06T01:41:13.330 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.329+0000 7fe305ffb700 1 -- 172.21.15.27:0/601325630 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 3 and data pool 4 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7fe300055a60 con 0x7fe30815dab0 2023-10-06T01:41:13.338 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.338+0000 7fe310095700 1 -- 172.21.15.27:0/601325630 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe2f4066f30 msgr2=0x7fe2f40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:13.338 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.338+0000 7fe310095700 1 --2- 172.21.15.27:0/601325630 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe2f4066f30 0x7fe2f40693e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe2f80091e0 tx=0x7fe2f8017000 comp rx=0 tx=0).stop 2023-10-06T01:41:13.338 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.338+0000 7fe310095700 1 -- 172.21.15.27:0/601325630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe30815dab0 msgr2=0x7fe3080b0320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:13.338 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.338+0000 7fe310095700 1 --2- 172.21.15.27:0/601325630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe30815dab0 0x7fe3080b0320 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fe300009d70 tx=0x7fe30000b040 comp rx=0 tx=0).stop 2023-10-06T01:41:13.340 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.339+0000 7fe310095700 1 -- 172.21.15.27:0/601325630 shutdown_connections 2023-10-06T01:41:13.340 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.339+0000 7fe310095700 1 --2- 172.21.15.27:0/601325630 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe2f4066f30 0x7fe2f40693e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:13.340 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.339+0000 7fe310095700 1 --2- 172.21.15.27:0/601325630 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe30815dab0 0x7fe3080b0320 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:13.340 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.340+0000 7fe310095700 1 --2- 172.21.15.27:0/601325630 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe3080b3fd0 0x7fe3081421f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:13.340 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.340+0000 7fe310095700 1 --2- 172.21.15.27:0/601325630 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe308159500 0x7fe30815b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:13.340 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.340+0000 7fe310095700 1 -- 172.21.15.27:0/601325630 >> 172.21.15.27:0/601325630 conn(0x7fe308067630 msgr2=0x7fe308157d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:13.341 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.341+0000 7fe310095700 1 -- 172.21.15.27:0/601325630 shutdown_connections 2023-10-06T01:41:13.341 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.341+0000 7fe310095700 1 -- 172.21.15.27:0/601325630 wait complete. 2023-10-06T01:41:13.344 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 3 and data pool 4 2023-10-06T01:41:13.363 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T01:41:13.483 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.483+0000 7effe85c8700 1 Processor -- start 2023-10-06T01:41:13.483 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.483+0000 7effe85c8700 1 -- start start 2023-10-06T01:41:13.489 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.483+0000 7effe85c8700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7effe00c8d60 0x7effe00cc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:13.489 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.484+0000 7effe85c8700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effe00cad80 con 0x7effe00c9130 2023-10-06T01:41:13.489 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.484+0000 7effe85c8700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effe00cd580 con 0x7effe015b170 2023-10-06T01:41:13.490 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.484+0000 7effe85c8700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effe00cc970 con 0x7effe00c8d60 2023-10-06T01:41:13.490 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.484+0000 7effe6b65700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7effe00c9130 0x7effe0158270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47328/0 (socket says 172.21.15.27:47328) 2023-10-06T01:41:13.490 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.484+0000 7effe6b65700 1 -- 172.21.15.27:0/2642023979 learned_addr learned my addr 172.21.15.27:0/2642023979 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:13.490 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.484+0000 7effe6364700 1 --2- 172.21.15.27:0/2642023979 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7effe00c8d60 0x7effe00cc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:13.490 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.484+0000 7effe5362700 1 -- 172.21.15.27:0/2642023979 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3547050727 0 0) 0x7effe00cad80 con 0x7effe00c9130 2023-10-06T01:41:13.490 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.484+0000 7effe5362700 1 -- 172.21.15.27:0/2642023979 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effc8003570 con 0x7effe00c9130 2023-10-06T01:41:13.491 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.484+0000 7effe5362700 1 -- 172.21.15.27:0/2642023979 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 686662988 0 0) 0x7effc8003570 con 0x7effe00c9130 2023-10-06T01:41:13.491 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.484+0000 7effe5362700 1 -- 172.21.15.27:0/2642023979 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7effe00c8d60 msgr2=0x7effe00cc2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:13.491 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.484+0000 7effe5362700 1 --2- 172.21.15.27:0/2642023979 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7effe00c8d60 0x7effe00cc2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:13.491 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.484+0000 7effe5362700 1 -- 172.21.15.27:0/2642023979 >> v1:172.21.15.124:6789/0 conn(0x7effe015b170 legacy=0x7effe00c3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:13.491 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.484+0000 7effe5362700 1 -- 172.21.15.27:0/2642023979 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe00ccc50 con 0x7effe00c9130 2023-10-06T01:41:13.491 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.484+0000 7effe5362700 1 -- 172.21.15.27:0/2642023979 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2619078944 0 0) 0x7effd8001470 con 0x7effe00c9130 2023-10-06T01:41:13.491 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.484+0000 7effe5362700 1 -- 172.21.15.27:0/2642023979 >> v1:172.21.15.27:6789/0 conn(0x7effe00c9130 legacy=0x7effe0158270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:13.491 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.484+0000 7effe5362700 1 --2- 172.21.15.27:0/2642023979 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7effc8003a90 0x7effc8003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:13.492 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.484+0000 7effe5362700 1 --2- 172.21.15.27:0/2642023979 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7effe00c8d60 0x7effc8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:13.492 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.484+0000 7effe5362700 1 --2- 172.21.15.27:0/2642023979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc8004d50 0x7effc80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:13.492 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.484+0000 7effe5362700 1 -- 172.21.15.27:0/2642023979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe00cc970 con 0x7effc8004d50 2023-10-06T01:41:13.492 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.484+0000 7effe5362700 1 -- 172.21.15.27:0/2642023979 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc800bb10 con 0x7effc8003a90 2023-10-06T01:41:13.492 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.484+0000 7effe5362700 1 -- 172.21.15.27:0/2642023979 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc800bcd0 con 0x7effe00c8d60 2023-10-06T01:41:13.492 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.484+0000 7effe6b65700 1 --2- 172.21.15.27:0/2642023979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc8004d50 0x7effc80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:13.492 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.484+0000 7effe6364700 1 --2- 172.21.15.27:0/2642023979 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7effc8003a90 0x7effc8003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:13.492 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.484+0000 7effe5b63700 1 --2- 172.21.15.27:0/2642023979 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7effe00c8d60 0x7effc8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:13.492 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.485+0000 7effe6b65700 1 -- 172.21.15.27:0/2642023979 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7effe00c8d60 msgr2=0x7effc8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:13.493 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.485+0000 7effe6b65700 1 --2- 172.21.15.27:0/2642023979 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7effe00c8d60 0x7effc8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:13.493 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.485+0000 7effe6b65700 1 -- 172.21.15.27:0/2642023979 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7effc8003a90 msgr2=0x7effc8003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:13.493 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.485+0000 7effe6b65700 1 --2- 172.21.15.27:0/2642023979 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7effc8003a90 0x7effc8003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:13.493 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.485+0000 7effe6b65700 1 -- 172.21.15.27:0/2642023979 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc8003570 con 0x7effc8004d50 2023-10-06T01:41:13.493 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.485+0000 7effe6b65700 1 --2- 172.21.15.27:0/2642023979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc8004d50 0x7effc80051c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7effd800c3c0 tx=0x7effd8001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd4b61ce6ada5443 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:13.493 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.485+0000 7effe5362700 1 -- 172.21.15.27:0/2642023979 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd800cc90 con 0x7effc8004d50 2023-10-06T01:41:13.493 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.485+0000 7effe5362700 1 -- 172.21.15.27:0/2642023979 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effd800cdf0 con 0x7effc8004d50 2023-10-06T01:41:13.493 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.485+0000 7effe5362700 1 -- 172.21.15.27:0/2642023979 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd8005b70 con 0x7effc8004d50 2023-10-06T01:41:13.493 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.485+0000 7effe85c8700 1 -- 172.21.15.27:0/2642023979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc8004d50 msgr2=0x7effc80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:13.494 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.485+0000 7effe85c8700 1 --2- 172.21.15.27:0/2642023979 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effc8004d50 0x7effc80051c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7effd800c3c0 tx=0x7effd8001e70 comp rx=0 tx=0).stop 2023-10-06T01:41:13.494 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.485+0000 7effe6b65700 1 -- 172.21.15.27:0/2642023979 reap_dead start 2023-10-06T01:41:13.494 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.486+0000 7effe85c8700 1 -- 172.21.15.27:0/2642023979 shutdown_connections 2023-10-06T01:41:13.494 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.486+0000 7effe85c8700 1 -- 172.21.15.27:0/2642023979 >> 172.21.15.27:0/2642023979 conn(0x7effe00ce4d0 msgr2=0x7effe00ce8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:13.494 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.486+0000 7effe85c8700 1 -- 172.21.15.27:0/2642023979 shutdown_connections 2023-10-06T01:41:13.494 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.486+0000 7effe85c8700 1 -- 172.21.15.27:0/2642023979 wait complete. 2023-10-06T01:41:13.494 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.487+0000 7effe85c8700 1 Processor -- start 2023-10-06T01:41:13.494 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.487+0000 7effe85c8700 1 -- start start 2023-10-06T01:41:13.495 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.487+0000 7effe85c8700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7effc8004d50 0x7effe01510c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:13.495 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.487+0000 7effe85c8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effe006a480 0x7effe00c6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:13.495 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.487+0000 7effe85c8700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7effe00c6c80 0x7effe015ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:13.495 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.487+0000 7effe85c8700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe0069ff0 con 0x7effe006a480 2023-10-06T01:41:13.495 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.487+0000 7effe85c8700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe00693e0 con 0x7effe00c6c80 2023-10-06T01:41:13.495 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.487+0000 7effe85c8700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effe00696c0 con 0x7effc8004d50 2023-10-06T01:41:13.495 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.488+0000 7effe5b63700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effe006a480 0x7effe00c6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:13.495 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.488+0000 7effe5b63700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effe006a480 0x7effe00c6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40352/0 (socket says 172.21.15.27:40352) 2023-10-06T01:41:13.495 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.488+0000 7effe5b63700 1 -- 172.21.15.27:0/2836448355 learned_addr learned my addr 172.21.15.27:0/2836448355 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:13.495 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.488+0000 7effe6364700 1 --2- 172.21.15.27:0/2836448355 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7effc8004d50 0x7effe01510c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:13.496 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.488+0000 7effe6b65700 1 --2- 172.21.15.27:0/2836448355 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7effe00c6c80 0x7effe015ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:13.496 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.488+0000 7effe5b63700 1 -- 172.21.15.27:0/2836448355 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7effc8004d50 msgr2=0x7effe01510c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:13.496 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.488+0000 7effe5b63700 1 --2- 172.21.15.27:0/2836448355 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7effc8004d50 0x7effe01510c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:13.496 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.488+0000 7effe5b63700 1 -- 172.21.15.27:0/2836448355 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7effe00c6c80 msgr2=0x7effe015ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:13.496 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.488+0000 7effe5b63700 1 --2- 172.21.15.27:0/2836448355 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7effe00c6c80 0x7effe015ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:13.496 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.488+0000 7effe5b63700 1 -- 172.21.15.27:0/2836448355 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe00699a0 con 0x7effe006a480 2023-10-06T01:41:13.496 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.488+0000 7effe5b63700 1 --2- 172.21.15.27:0/2836448355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effe006a480 0x7effe00c6740 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7effd4002ab0 tx=0x7effd4007d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:13.496 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.488+0000 7effde7fc700 1 -- 172.21.15.27:0/2836448355 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd400f8d0 con 0x7effe006a480 2023-10-06T01:41:13.496 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.488+0000 7effde7fc700 1 -- 172.21.15.27:0/2836448355 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effd400fa30 con 0x7effe006a480 2023-10-06T01:41:13.496 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.488+0000 7effe85c8700 1 -- 172.21.15.27:0/2836448355 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effe0069c80 con 0x7effe006a480 2023-10-06T01:41:13.496 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.488+0000 7effde7fc700 1 -- 172.21.15.27:0/2836448355 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd400fe30 con 0x7effe006a480 2023-10-06T01:41:13.496 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.488+0000 7effe85c8700 1 -- 172.21.15.27:0/2836448355 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effe0069000 con 0x7effe006a480 2023-10-06T01:41:13.497 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.489+0000 7effe85c8700 1 -- 172.21.15.27:0/2836448355 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effe0064050 con 0x7effe006a480 2023-10-06T01:41:13.497 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.490+0000 7effde7fc700 1 -- 172.21.15.27:0/2836448355 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7effd4013600 con 0x7effe006a480 2023-10-06T01:41:13.497 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.490+0000 7effde7fc700 1 --2- 172.21.15.27:0/2836448355 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7effcc066f30 0x7effcc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.490+0000 7effde7fc700 1 -- 172.21.15.27:0/2836448355 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 6058+0+0 (secure 0 0 0) 0x7effd4084bb0 con 0x7effe006a480 2023-10-06T01:41:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.490+0000 7effe6364700 1 --2- 172.21.15.27:0/2836448355 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7effcc066f30 0x7effcc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.491+0000 7effe6364700 1 --2- 172.21.15.27:0/2836448355 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7effcc066f30 0x7effcc0693e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7effd00094d0 tx=0x7effd000b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:13.498 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.495+0000 7effde7fc700 1 -- 172.21.15.27:0/2836448355 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7effd4051d70 con 0x7effe006a480 2023-10-06T01:41:13.690 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.690+0000 7effe85c8700 1 -- 172.21.15.27:0/2836448355 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7effe00ca4f0 con 0x7effe006a480 2023-10-06T01:41:13.691 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.691+0000 7effde7fc700 1 -- 172.21.15.27:0/2836448355 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16879 (secure 0 0 0) 0x7effd4017090 con 0x7effe006a480 2023-10-06T01:41:13.691 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:13.691 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"fsid":"20a7bff0-74f8-4f2f-b6c6-899e804bc965","created":"2023-10-06T01:40:54.474215+0000","modified":"2023-10-06T01:41:12.735646+0000","last_up_change":"2023-10-06T01:41:05.242847+0000","last_in_change":"2023-10-06T01:40:56.890960+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-10-06T01:41:05.685386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-10-06T01:41:06.092995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T01:41:10.434257+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T01:41:11.700262+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b902c86d-800c-40c5-b152-b545e50d02c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":90644},{"type":"v1","addr":"172.21.15.27:6801","nonce":90644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":90644},{"type":"v1","addr":"172.21.15.27:6803","nonce":90644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":90644},{"type":"v1","addr":"172.21.15.27:6807","nonce":90644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":90644},{"type":"v1","addr":"172.21.15.27:6805","nonce":90644}]},"public_addr":"172.21.15.27:6801/90644","cluster_addr":"172.21.15.27:6803/90644","heartbeat_back_addr":"172.21.15.27:6807/90644","heartbeat_front_addr":"172.21.15.27:6805/90644","state":["exists","up"]},{"osd":1,"uuid":"5b4a70e5-b06e-48cd-bfc4-7962c5d05386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":90643},{"type":"v1","addr":"172.21.15.27:6809","nonce":90643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":90643},{"type":"v1","addr":"172.21.15.27:6811","nonce":90643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":90643},{"type":"v1","addr":"172.21.15.27:6815","nonce":90643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":90643},{"type":"v1","addr":"172.21.15.27:6813","nonce":90643}]},"public_addr":"172.21.15.27:6809/90643","cluster_addr":"172.21.15.27:6811/90643","heartbeat_back_addr":"172.21.15.27:6815/90643","heartbeat_front_addr":"172.21.15.27:6813/90643","state":["exists","up"]},{"osd":2,"uuid":"22d9de46-4073-4336-9122-5227e601eca0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":90645},{"type":"v1","addr":"172.21.15.27:6817","nonce":90645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":90645},{"type":"v1","addr":"172.21.15.27:6819","nonce":90645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":90645},{"type":"v1","addr":"172.21.15.27:6823","nonce":90645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":90645},{"type":"v1","addr":"172.21.15.27:6821","nonce":90645}]},"public_addr":"172.21.15.27:6817/90645","cluster_addr":"172.21.15.27:6819/90645","heartbeat_back_addr":"172.21.15.27:6823/90645","heartbeat_front_addr":"172.21.15.27:6821/90645","state":["exists","up"]},{"osd":3,"uuid":"438bb2d3-34e2-4b7c-9734-8360c1b67650","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":90668},{"type":"v1","addr":"172.21.15.27:6825","nonce":90668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":90668},{"type":"v1","addr":"172.21.15.27:6827","nonce":90668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":90668},{"type":"v1","addr":"172.21.15.27:6831","nonce":90668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":90668},{"type":"v1","addr":"172.21.15.27:6829","nonce":90668}]},"public_addr":"172.21.15.27:6825/90668","cluster_addr":"172.21.15.27:6827/90668","heartbeat_back_addr":"172.21.15.27:6831/90668","heartbeat_front_addr":"172.21.15.27:6829/90668","state":["exists","up"]},{"osd":4,"uuid":"e1956a8e-e0f8-4e7c-acbb-40037415a354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":90863},{"type":"v1","addr":"172.21.15.124:6801","nonce":90863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":90863},{"type":"v1","addr":"172.21.15.124:6803","nonce":90863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":90863},{"type":"v1","addr":"172.21.15.124:6807","nonce":90863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":90863},{"type":"v1","addr":"172.21.15.124:6805","nonce":90863}]},"public_addr":"172.21.15.124:6801/90863","cluster_addr":"172.21.15.124:6803/90863","heartbeat_back_addr":"172.21.15.124:6807/90863","heartbeat_front_addr":"172.21.15.124:6805/90863","state":["exists","up"]},{"osd":5,"uuid":"306c59d4-846d-4b73-b1c9-051bec9e9368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":90885},{"type":"v1","addr":"172.21.15.124:6817","nonce":90885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":90885},{"type":"v1","addr":"172.21.15.124:6819","nonce":90885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":90885},{"type":"v1","addr":"172.21.15.124:6823","nonce":90885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":90885},{"type":"v1","addr":"172.21.15.124:6821","nonce":90885}]},"public_addr":"172.21.15.124:6817/90885","cluster_addr":"172.21.15.124:6819/90885","heartbeat_back_addr":"172.21.15.124:6823/90885","heartbeat_front_addr":"172.21.15.124:6821/90885","state":["exists","up"]},{"osd":6,"uuid":"15463e53-ac8d-47d1-b4f6-c0d31f29e4a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":90864},{"type":"v1","addr":"172.21.15.124:6809","nonce":90864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":90864},{"type":"v1","addr":"172.21.15.124:6811","nonce":90864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":90864},{"type":"v1","addr":"172.21.15.124:6815","nonce":90864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":90864},{"type":"v1","addr":"172.21.15.124:6813","nonce":90864}]},"public_addr":"172.21.15.124:6809/90864","cluster_addr":"172.21.15.124:6811/90864","heartbeat_back_addr":"172.21.15.124:6815/90864","heartbeat_front_addr":"172.21.15.124:6813/90864","state":["exists","up"]},{"osd":7,"uuid":"ae463a90-a9d3-498f-86fa-47a35b3fe889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":90894},{"type":"v1","addr":"172.21.15.124:6825","nonce":90894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":90894},{"type":"v1","addr":"172.21.15.124:6827","nonce":90894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":90894},{"type":"v1","addr":"172.21.15.124:6831","nonce":90894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":90894},{"type":"v1","addr":"172.21.15.124:6829","nonce":90894}]},"public_addr":"172.21.15.124:6825/90894","cluster_addr":"172.21.15.124:6827/90894","heartbeat_back_addr":"172.21.15.124:6831/90894","heartbeat_front_addr":"172.21.15.124:6829/90894","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.827704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.839951+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.032230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.050605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.987908+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.031290+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.080707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.053692+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T01:41:13.693 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.693+0000 7effe85c8700 1 -- 172.21.15.27:0/2836448355 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7effcc066f30 msgr2=0x7effcc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:13.694 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.693+0000 7effe85c8700 1 --2- 172.21.15.27:0/2836448355 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7effcc066f30 0x7effcc0693e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7effd00094d0 tx=0x7effd000b040 comp rx=0 tx=0).stop 2023-10-06T01:41:13.694 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.693+0000 7effe85c8700 1 -- 172.21.15.27:0/2836448355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effe006a480 msgr2=0x7effe00c6740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:13.694 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.693+0000 7effe85c8700 1 --2- 172.21.15.27:0/2836448355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effe006a480 0x7effe00c6740 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7effd4002ab0 tx=0x7effd4007d10 comp rx=0 tx=0).stop 2023-10-06T01:41:13.694 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.693+0000 7effe85c8700 1 -- 172.21.15.27:0/2836448355 shutdown_connections 2023-10-06T01:41:13.694 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.693+0000 7effe85c8700 1 --2- 172.21.15.27:0/2836448355 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7effcc066f30 0x7effcc0693e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:13.694 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.693+0000 7effe85c8700 1 --2- 172.21.15.27:0/2836448355 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7effc8004d50 0x7effe01510c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:13.694 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.693+0000 7effe85c8700 1 --2- 172.21.15.27:0/2836448355 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7effe00c6c80 0x7effe015ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:13.694 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.693+0000 7effe85c8700 1 --2- 172.21.15.27:0/2836448355 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7effe006a480 0x7effe00c6740 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:13.694 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.693+0000 7effe85c8700 1 -- 172.21.15.27:0/2836448355 >> 172.21.15.27:0/2836448355 conn(0x7effe00ce4d0 msgr2=0x7effe0151e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:13.694 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.693+0000 7effe85c8700 1 -- 172.21.15.27:0/2836448355 shutdown_connections 2023-10-06T01:41:13.694 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.693+0000 7effe85c8700 1 -- 172.21.15.27:0/2836448355 wait complete. 2023-10-06T01:41:13.705 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T01:41:13.823 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.823+0000 7fe3b3548700 1 Processor -- start 2023-10-06T01:41:13.824 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.823+0000 7fe3b3548700 1 -- start start 2023-10-06T01:41:13.826 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.824+0000 7fe3b3548700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe3ac0b4060 0x7fe3ac158560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:13.826 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.824+0000 7fe3b3548700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe3ac0add70 0x7fe3ac0ae190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:13.826 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.824+0000 7fe3b3548700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3ac1580d0 con 0x7fe3ac157570 2023-10-06T01:41:13.826 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.824+0000 7fe3b3548700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3ac0ae6d0 con 0x7fe3ac0add70 2023-10-06T01:41:13.826 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.824+0000 7fe3b3548700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3ac15ba40 con 0x7fe3ac0b4060 2023-10-06T01:41:13.826 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.824+0000 7fe3b1ae5700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe3ac0add70 0x7fe3ac0ae190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:13.826 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.824+0000 7fe3b1ae5700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe3ac0add70 0x7fe3ac0ae190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40356/0 (socket says 172.21.15.27:40356) 2023-10-06T01:41:13.827 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.824+0000 7fe3b12e4700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe3ac157570 0x7fe3ac15b210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47336/0 (socket says 172.21.15.27:47336) 2023-10-06T01:41:13.827 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.824+0000 7fe3b1ae5700 1 -- 172.21.15.27:0/1739161717 learned_addr learned my addr 172.21.15.27:0/1739161717 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:13.827 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.824+0000 7fe3b0ae3700 1 --2- 172.21.15.27:0/1739161717 >> v2:172.21.15.124:3300/0 conn(0x7fe3ac0b4060 0x7fe3ac158560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:13.827 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.824+0000 7fe3b1ae5700 1 -- 172.21.15.27:0/1739161717 >> v2:172.21.15.124:3300/0 conn(0x7fe3ac0b4060 msgr2=0x7fe3ac158560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:13.827 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.824+0000 7fe3b1ae5700 1 --2- 172.21.15.27:0/1739161717 >> v2:172.21.15.124:3300/0 conn(0x7fe3ac0b4060 0x7fe3ac158560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:13.827 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.825+0000 7fe3b1ae5700 1 -- 172.21.15.27:0/1739161717 >> v1:172.21.15.27:6789/0 conn(0x7fe3ac157570 legacy=0x7fe3ac15b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:13.827 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.825+0000 7fe3b1ae5700 1 -- 172.21.15.27:0/1739161717 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3ac15bd20 con 0x7fe3ac0add70 2023-10-06T01:41:13.827 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.825+0000 7fe3b0ae3700 1 --2- 172.21.15.27:0/1739161717 >> v2:172.21.15.124:3300/0 conn(0x7fe3ac0b4060 0x7fe3ac158560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:13.827 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.825+0000 7fe3b1ae5700 1 --2- 172.21.15.27:0/1739161717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3ac0add70 0x7fe3ac0ae190 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac06f220 tx=0x7fe3a4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57de68293598b93a server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:13.827 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.825+0000 7fe3a3fff700 1 -- 172.21.15.27:0/1739161717 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a400e070 con 0x7fe3ac0add70 2023-10-06T01:41:13.827 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.825+0000 7fe3a3fff700 1 -- 172.21.15.27:0/1739161717 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3a400ba50 con 0x7fe3ac0add70 2023-10-06T01:41:13.827 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.825+0000 7fe3a3fff700 1 -- 172.21.15.27:0/1739161717 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a40092e0 con 0x7fe3ac0add70 2023-10-06T01:41:13.827 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.825+0000 7fe3b3548700 1 -- 172.21.15.27:0/1739161717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3ac0add70 msgr2=0x7fe3ac0ae190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:13.828 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.825+0000 7fe3b3548700 1 --2- 172.21.15.27:0/1739161717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3ac0add70 0x7fe3ac0ae190 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac06f220 tx=0x7fe3a4009840 comp rx=0 tx=0).stop 2023-10-06T01:41:13.828 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.825+0000 7fe3b3548700 1 -- 172.21.15.27:0/1739161717 shutdown_connections 2023-10-06T01:41:13.828 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.825+0000 7fe3b3548700 1 --2- 172.21.15.27:0/1739161717 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3ac0add70 0x7fe3ac0ae190 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:13.828 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.825+0000 7fe3b3548700 1 --2- 172.21.15.27:0/1739161717 >> v2:172.21.15.124:3300/0 conn(0x7fe3ac0b4060 0x7fe3ac158560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:13.828 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.825+0000 7fe3b3548700 1 -- 172.21.15.27:0/1739161717 >> 172.21.15.27:0/1739161717 conn(0x7fe3ac067630 msgr2=0x7fe3ac067a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:13.828 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.825+0000 7fe3b3548700 1 -- 172.21.15.27:0/1739161717 shutdown_connections 2023-10-06T01:41:13.828 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.825+0000 7fe3b3548700 1 -- 172.21.15.27:0/1739161717 wait complete. 2023-10-06T01:41:13.830 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.826+0000 7fe3b3548700 1 Processor -- start 2023-10-06T01:41:13.831 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.827+0000 7fe3b3548700 1 -- start start 2023-10-06T01:41:13.831 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.827+0000 7fe3b3548700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe3ac0add70 0x7fe3ac0c62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:13.831 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.827+0000 7fe3b3548700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe3ac0b4060 0x7fe3ac153bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:13.831 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.827+0000 7fe3b3548700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3ac157570 0x7fe3ac0c3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:13.831 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.827+0000 7fe3b3548700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ac167390 con 0x7fe3ac157570 2023-10-06T01:41:13.831 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.827+0000 7fe3b3548700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ac0c4480 con 0x7fe3ac0b4060 2023-10-06T01:41:13.831 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.827+0000 7fe3b3548700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3ac0c4760 con 0x7fe3ac0add70 2023-10-06T01:41:13.831 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.827+0000 7fe3b1ae5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3ac157570 0x7fe3ac0c3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:13.831 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.827+0000 7fe3b1ae5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3ac157570 0x7fe3ac0c3be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40362/0 (socket says 172.21.15.27:40362) 2023-10-06T01:41:13.831 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.827+0000 7fe3b1ae5700 1 -- 172.21.15.27:0/2820849458 learned_addr learned my addr 172.21.15.27:0/2820849458 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:13.831 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.828+0000 7fe3b12e4700 1 --2- 172.21.15.27:0/2820849458 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe3ac0add70 0x7fe3ac0c62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:13.832 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.828+0000 7fe3b0ae3700 1 --2- 172.21.15.27:0/2820849458 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe3ac0b4060 0x7fe3ac153bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:13.832 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.828+0000 7fe3b1ae5700 1 -- 172.21.15.27:0/2820849458 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe3ac0add70 msgr2=0x7fe3ac0c62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:13.832 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.828+0000 7fe3b1ae5700 1 --2- 172.21.15.27:0/2820849458 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe3ac0add70 0x7fe3ac0c62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:13.832 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.828+0000 7fe3b1ae5700 1 -- 172.21.15.27:0/2820849458 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe3ac0b4060 msgr2=0x7fe3ac153bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:13.832 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.828+0000 7fe3b1ae5700 1 --2- 172.21.15.27:0/2820849458 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe3ac0b4060 0x7fe3ac153bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:13.832 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.828+0000 7fe3b1ae5700 1 -- 172.21.15.27:0/2820849458 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a4009ae0 con 0x7fe3ac157570 2023-10-06T01:41:13.832 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.828+0000 7fe3b12e4700 1 --2- 172.21.15.27:0/2820849458 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe3ac0add70 0x7fe3ac0c62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:13.832 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.828+0000 7fe3b1ae5700 1 --2- 172.21.15.27:0/2820849458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3ac157570 0x7fe3ac0c3be0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac064a60 tx=0x7fe3a4018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:13.832 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.828+0000 7fe3a17fa700 1 -- 172.21.15.27:0/2820849458 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a4008e40 con 0x7fe3ac157570 2023-10-06T01:41:13.832 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.828+0000 7fe3a17fa700 1 -- 172.21.15.27:0/2820849458 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3a400be60 con 0x7fe3ac157570 2023-10-06T01:41:13.832 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.828+0000 7fe3b3548700 1 -- 172.21.15.27:0/2820849458 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3ac0c4a40 con 0x7fe3ac157570 2023-10-06T01:41:13.832 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.828+0000 7fe3a17fa700 1 -- 172.21.15.27:0/2820849458 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a40266b0 con 0x7fe3ac157570 2023-10-06T01:41:13.832 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.828+0000 7fe3b3548700 1 -- 172.21.15.27:0/2820849458 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3ac1540f0 con 0x7fe3ac157570 2023-10-06T01:41:13.833 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.829+0000 7fe3b3548700 1 -- 172.21.15.27:0/2820849458 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3ac167540 con 0x7fe3ac157570 2023-10-06T01:41:13.833 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.829+0000 7fe3a17fa700 1 -- 172.21.15.27:0/2820849458 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe3a4003040 con 0x7fe3ac157570 2023-10-06T01:41:13.833 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.830+0000 7fe3a17fa700 1 --2- 172.21.15.27:0/2820849458 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe390066f60 0x7fe390069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:13.833 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.830+0000 7fe3a17fa700 1 -- 172.21.15.27:0/2820849458 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fe3a40865e0 con 0x7fe3ac157570 2023-10-06T01:41:13.833 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.830+0000 7fe3b12e4700 1 --2- 172.21.15.27:0/2820849458 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe390066f60 0x7fe390069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:13.833 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.830+0000 7fe3b12e4700 1 --2- 172.21.15.27:0/2820849458 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe390066f60 0x7fe390069410 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe3a8007fd0 tx=0x7fe3a8007e10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:13.834 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:13.834+0000 7fe3a17fa700 1 -- 172.21.15.27:0/2820849458 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe3ac167540 con 0x7fe3ac157570 2023-10-06T01:41:14.026 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.026+0000 7fe3b3548700 1 -- 172.21.15.27:0/2820849458 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3ac064050 con 0x7fe3ac157570 2023-10-06T01:41:14.027 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.027+0000 7fe3a17fa700 1 -- 172.21.15.27:0/2820849458 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16879 (secure 0 0 0) 0x7fe3a40536f0 con 0x7fe3ac157570 2023-10-06T01:41:14.028 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:14.028 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":19,"fsid":"20a7bff0-74f8-4f2f-b6c6-899e804bc965","created":"2023-10-06T01:40:54.474215+0000","modified":"2023-10-06T01:41:12.735646+0000","last_up_change":"2023-10-06T01:41:05.242847+0000","last_in_change":"2023-10-06T01:40:56.890960+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-10-06T01:41:05.685386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-10-06T01:41:06.092995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T01:41:10.434257+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T01:41:11.700262+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b902c86d-800c-40c5-b152-b545e50d02c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":90644},{"type":"v1","addr":"172.21.15.27:6801","nonce":90644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":90644},{"type":"v1","addr":"172.21.15.27:6803","nonce":90644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":90644},{"type":"v1","addr":"172.21.15.27:6807","nonce":90644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":90644},{"type":"v1","addr":"172.21.15.27:6805","nonce":90644}]},"public_addr":"172.21.15.27:6801/90644","cluster_addr":"172.21.15.27:6803/90644","heartbeat_back_addr":"172.21.15.27:6807/90644","heartbeat_front_addr":"172.21.15.27:6805/90644","state":["exists","up"]},{"osd":1,"uuid":"5b4a70e5-b06e-48cd-bfc4-7962c5d05386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":90643},{"type":"v1","addr":"172.21.15.27:6809","nonce":90643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":90643},{"type":"v1","addr":"172.21.15.27:6811","nonce":90643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":90643},{"type":"v1","addr":"172.21.15.27:6815","nonce":90643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":90643},{"type":"v1","addr":"172.21.15.27:6813","nonce":90643}]},"public_addr":"172.21.15.27:6809/90643","cluster_addr":"172.21.15.27:6811/90643","heartbeat_back_addr":"172.21.15.27:6815/90643","heartbeat_front_addr":"172.21.15.27:6813/90643","state":["exists","up"]},{"osd":2,"uuid":"22d9de46-4073-4336-9122-5227e601eca0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":90645},{"type":"v1","addr":"172.21.15.27:6817","nonce":90645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":90645},{"type":"v1","addr":"172.21.15.27:6819","nonce":90645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":90645},{"type":"v1","addr":"172.21.15.27:6823","nonce":90645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":90645},{"type":"v1","addr":"172.21.15.27:6821","nonce":90645}]},"public_addr":"172.21.15.27:6817/90645","cluster_addr":"172.21.15.27:6819/90645","heartbeat_back_addr":"172.21.15.27:6823/90645","heartbeat_front_addr":"172.21.15.27:6821/90645","state":["exists","up"]},{"osd":3,"uuid":"438bb2d3-34e2-4b7c-9734-8360c1b67650","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":90668},{"type":"v1","addr":"172.21.15.27:6825","nonce":90668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":90668},{"type":"v1","addr":"172.21.15.27:6827","nonce":90668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":90668},{"type":"v1","addr":"172.21.15.27:6831","nonce":90668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":90668},{"type":"v1","addr":"172.21.15.27:6829","nonce":90668}]},"public_addr":"172.21.15.27:6825/90668","cluster_addr":"172.21.15.27:6827/90668","heartbeat_back_addr":"172.21.15.27:6831/90668","heartbeat_front_addr":"172.21.15.27:6829/90668","state":["exists","up"]},{"osd":4,"uuid":"e1956a8e-e0f8-4e7c-acbb-40037415a354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":90863},{"type":"v1","addr":"172.21.15.124:6801","nonce":90863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":90863},{"type":"v1","addr":"172.21.15.124:6803","nonce":90863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":90863},{"type":"v1","addr":"172.21.15.124:6807","nonce":90863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":90863},{"type":"v1","addr":"172.21.15.124:6805","nonce":90863}]},"public_addr":"172.21.15.124:6801/90863","cluster_addr":"172.21.15.124:6803/90863","heartbeat_back_addr":"172.21.15.124:6807/90863","heartbeat_front_addr":"172.21.15.124:6805/90863","state":["exists","up"]},{"osd":5,"uuid":"306c59d4-846d-4b73-b1c9-051bec9e9368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":90885},{"type":"v1","addr":"172.21.15.124:6817","nonce":90885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":90885},{"type":"v1","addr":"172.21.15.124:6819","nonce":90885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":90885},{"type":"v1","addr":"172.21.15.124:6823","nonce":90885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":90885},{"type":"v1","addr":"172.21.15.124:6821","nonce":90885}]},"public_addr":"172.21.15.124:6817/90885","cluster_addr":"172.21.15.124:6819/90885","heartbeat_back_addr":"172.21.15.124:6823/90885","heartbeat_front_addr":"172.21.15.124:6821/90885","state":["exists","up"]},{"osd":6,"uuid":"15463e53-ac8d-47d1-b4f6-c0d31f29e4a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":90864},{"type":"v1","addr":"172.21.15.124:6809","nonce":90864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":90864},{"type":"v1","addr":"172.21.15.124:6811","nonce":90864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":90864},{"type":"v1","addr":"172.21.15.124:6815","nonce":90864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":90864},{"type":"v1","addr":"172.21.15.124:6813","nonce":90864}]},"public_addr":"172.21.15.124:6809/90864","cluster_addr":"172.21.15.124:6811/90864","heartbeat_back_addr":"172.21.15.124:6815/90864","heartbeat_front_addr":"172.21.15.124:6813/90864","state":["exists","up"]},{"osd":7,"uuid":"ae463a90-a9d3-498f-86fa-47a35b3fe889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":90894},{"type":"v1","addr":"172.21.15.124:6825","nonce":90894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":90894},{"type":"v1","addr":"172.21.15.124:6827","nonce":90894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":90894},{"type":"v1","addr":"172.21.15.124:6831","nonce":90894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":90894},{"type":"v1","addr":"172.21.15.124:6829","nonce":90894}]},"public_addr":"172.21.15.124:6825/90894","cluster_addr":"172.21.15.124:6827/90894","heartbeat_back_addr":"172.21.15.124:6831/90894","heartbeat_front_addr":"172.21.15.124:6829/90894","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.827704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.839951+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.032230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.050605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.987908+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.031290+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.080707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.053692+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T01:41:14.030 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.029+0000 7fe3b3548700 1 -- 172.21.15.27:0/2820849458 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe390066f60 msgr2=0x7fe390069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:14.030 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.029+0000 7fe3b3548700 1 --2- 172.21.15.27:0/2820849458 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe390066f60 0x7fe390069410 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe3a8007fd0 tx=0x7fe3a8007e10 comp rx=0 tx=0).stop 2023-10-06T01:41:14.030 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.029+0000 7fe3b3548700 1 -- 172.21.15.27:0/2820849458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3ac157570 msgr2=0x7fe3ac0c3be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:14.030 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.029+0000 7fe3b3548700 1 --2- 172.21.15.27:0/2820849458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3ac157570 0x7fe3ac0c3be0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac064a60 tx=0x7fe3a4018000 comp rx=0 tx=0).stop 2023-10-06T01:41:14.030 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.030+0000 7fe3b3548700 1 -- 172.21.15.27:0/2820849458 shutdown_connections 2023-10-06T01:41:14.030 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.030+0000 7fe3b3548700 1 --2- 172.21.15.27:0/2820849458 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe390066f60 0x7fe390069410 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:14.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.030+0000 7fe3b3548700 1 --2- 172.21.15.27:0/2820849458 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe3ac0add70 0x7fe3ac0c62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:14.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.030+0000 7fe3b3548700 1 --2- 172.21.15.27:0/2820849458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe3ac157570 0x7fe3ac0c3be0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:14.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.030+0000 7fe3b3548700 1 --2- 172.21.15.27:0/2820849458 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe3ac0b4060 0x7fe3ac153bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:14.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.030+0000 7fe3b3548700 1 -- 172.21.15.27:0/2820849458 >> 172.21.15.27:0/2820849458 conn(0x7fe3ac067630 msgr2=0x7fe3ac0cbfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:14.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.030+0000 7fe3b3548700 1 -- 172.21.15.27:0/2820849458 shutdown_connections 2023-10-06T01:41:14.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.030+0000 7fe3b3548700 1 -- 172.21.15.27:0/2820849458 wait complete. 2023-10-06T01:41:14.042 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-06T01:41:14.163 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.163+0000 7f1ffa7a7700 1 Processor -- start 2023-10-06T01:41:14.164 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.163+0000 7f1ffa7a7700 1 -- start start 2023-10-06T01:41:14.167 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.164+0000 7f1ffa7a7700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1ff40b4000 0x7f1ff4159b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:14.167 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.164+0000 7f1ffa7a7700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f1ff40b1690 0x7f1ff40b1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:14.167 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.164+0000 7f1ffa7a7700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff40698e0 con 0x7f1ff415cb10 2023-10-06T01:41:14.167 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.164+0000 7f1ffa7a7700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ff40b2020 con 0x7f1ff40b1690 2023-10-06T01:41:14.167 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.164+0000 7f1ffa7a7700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff4068fd0 con 0x7f1ff40b4000 2023-10-06T01:41:14.167 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.164+0000 7f1ff3fff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f1ff415cb10 0x7f1ff40af780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47338/0 (socket says 172.21.15.27:47338) 2023-10-06T01:41:14.168 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.164+0000 7f1ff3fff700 1 -- 172.21.15.27:0/1586490085 learned_addr learned my addr 172.21.15.27:0/1586490085 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:14.168 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.164+0000 7f1ff8d44700 1 --2- 172.21.15.27:0/1586490085 >> v2:172.21.15.124:3300/0 conn(0x7f1ff40b1690 0x7f1ff40b1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:14.168 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.164+0000 7f1ff37fe700 1 --2- 172.21.15.27:0/1586490085 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1ff40b4000 0x7f1ff4159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:14.168 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.164+0000 7f1ff2ffd700 1 -- 172.21.15.27:0/1586490085 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2039637854 0 0) 0x7f1ff40698e0 con 0x7f1ff415cb10 2023-10-06T01:41:14.168 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.164+0000 7f1ff2ffd700 1 -- 172.21.15.27:0/1586490085 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fdc003570 con 0x7f1ff415cb10 2023-10-06T01:41:14.168 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.165+0000 7f1ff2ffd700 1 -- 172.21.15.27:0/1586490085 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1722713585 0 0) 0x7f1fdc003570 con 0x7f1ff415cb10 2023-10-06T01:41:14.168 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.165+0000 7f1ff2ffd700 1 -- 172.21.15.27:0/1586490085 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1ff40b4000 msgr2=0x7f1ff4159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:14.168 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.165+0000 7f1ff2ffd700 1 --2- 172.21.15.27:0/1586490085 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1ff40b4000 0x7f1ff4159b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:14.168 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.165+0000 7f1ff2ffd700 1 -- 172.21.15.27:0/1586490085 >> v2:172.21.15.124:3300/0 conn(0x7f1ff40b1690 msgr2=0x7f1ff40b1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:14.168 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.165+0000 7f1ff2ffd700 1 --2- 172.21.15.27:0/1586490085 >> v2:172.21.15.124:3300/0 conn(0x7f1ff40b1690 0x7f1ff40b1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:14.168 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.165+0000 7f1ff2ffd700 1 -- 172.21.15.27:0/1586490085 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff40b0020 con 0x7f1ff415cb10 2023-10-06T01:41:14.168 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.165+0000 7f1ff2ffd700 1 -- 172.21.15.27:0/1586490085 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2619078944 0 0) 0x7f1fe8001470 con 0x7f1ff415cb10 2023-10-06T01:41:14.169 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.165+0000 7f1ff2ffd700 1 -- 172.21.15.27:0/1586490085 >> v1:172.21.15.27:6789/0 conn(0x7f1ff415cb10 legacy=0x7f1ff40af780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:14.169 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.165+0000 7f1ff2ffd700 1 --2- 172.21.15.27:0/1586490085 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1fdc003a60 0x7f1fdc003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:14.169 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.165+0000 7f1ff2ffd700 1 --2- 172.21.15.27:0/1586490085 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1fdc004810 0x7f1fdc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:14.169 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.165+0000 7f1ff2ffd700 1 --2- 172.21.15.27:0/1586490085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fdc0051c0 0x7f1fdc00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:14.169 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.165+0000 7f1ff2ffd700 1 -- 172.21.15.27:0/1586490085 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff40b2020 con 0x7f1fdc0051c0 2023-10-06T01:41:14.169 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.165+0000 7f1ff2ffd700 1 -- 172.21.15.27:0/1586490085 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff4068fd0 con 0x7f1fdc003a60 2023-10-06T01:41:14.169 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.165+0000 7f1ff2ffd700 1 -- 172.21.15.27:0/1586490085 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fdc00bfa0 con 0x7f1fdc004810 2023-10-06T01:41:14.169 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.165+0000 7f1ff37fe700 1 --2- 172.21.15.27:0/1586490085 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1ff40b4000 0x7f1ff4159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:41:14.169 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.165+0000 7f1ff8d44700 1 --2- 172.21.15.27:0/1586490085 >> v2:172.21.15.124:3300/0 conn(0x7f1ff40b1690 0x7f1ff40b1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:41:14.169 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.165+0000 7f1ff8d44700 1 --2- 172.21.15.27:0/1586490085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fdc0051c0 0x7f1fdc00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:14.169 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.165+0000 7f1ff3fff700 1 --2- 172.21.15.27:0/1586490085 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1fdc003a60 0x7f1fdc003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:14.169 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.165+0000 7f1ff37fe700 1 --2- 172.21.15.27:0/1586490085 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1fdc004810 0x7f1fdc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:14.170 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.165+0000 7f1ff8d44700 1 -- 172.21.15.27:0/1586490085 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1fdc004810 msgr2=0x7f1fdc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:14.170 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.165+0000 7f1ff8d44700 1 --2- 172.21.15.27:0/1586490085 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1fdc004810 0x7f1fdc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:14.170 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.165+0000 7f1ff8d44700 1 -- 172.21.15.27:0/1586490085 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1fdc003a60 msgr2=0x7f1fdc003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:14.170 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.165+0000 7f1ff8d44700 1 --2- 172.21.15.27:0/1586490085 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1fdc003a60 0x7f1fdc003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:14.170 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.165+0000 7f1ff8d44700 1 -- 172.21.15.27:0/1586490085 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fdc003570 con 0x7f1fdc0051c0 2023-10-06T01:41:14.170 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.165+0000 7f1ff8d44700 1 --2- 172.21.15.27:0/1586490085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fdc0051c0 0x7f1fdc00ba60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1fec00e3c0 tx=0x7f1fec00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a53fe809783f0f3 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:14.170 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.166+0000 7f1ff2ffd700 1 -- 172.21.15.27:0/1586490085 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fec00ec70 con 0x7f1fdc0051c0 2023-10-06T01:41:14.170 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.166+0000 7f1ff2ffd700 1 -- 172.21.15.27:0/1586490085 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fec00f040 con 0x7f1fdc0051c0 2023-10-06T01:41:14.170 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.166+0000 7f1ff2ffd700 1 -- 172.21.15.27:0/1586490085 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fec014480 con 0x7f1fdc0051c0 2023-10-06T01:41:14.170 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.166+0000 7f1ffa7a7700 1 -- 172.21.15.27:0/1586490085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fdc0051c0 msgr2=0x7f1fdc00ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:14.170 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.166+0000 7f1ffa7a7700 1 --2- 172.21.15.27:0/1586490085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1fdc0051c0 0x7f1fdc00ba60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f1fec00e3c0 tx=0x7f1fec00b040 comp rx=0 tx=0).stop 2023-10-06T01:41:14.170 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.166+0000 7f1ff8d44700 1 -- 172.21.15.27:0/1586490085 reap_dead start 2023-10-06T01:41:14.170 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.166+0000 7f1ffa7a7700 1 -- 172.21.15.27:0/1586490085 shutdown_connections 2023-10-06T01:41:14.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.166+0000 7f1ffa7a7700 1 -- 172.21.15.27:0/1586490085 >> 172.21.15.27:0/1586490085 conn(0x7f1ff406d890 msgr2=0x7f1ff406dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:14.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.166+0000 7f1ffa7a7700 1 -- 172.21.15.27:0/1586490085 shutdown_connections 2023-10-06T01:41:14.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.166+0000 7f1ffa7a7700 1 -- 172.21.15.27:0/1586490085 wait complete. 2023-10-06T01:41:14.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.167+0000 7f1ffa7a7700 1 Processor -- start 2023-10-06T01:41:14.173 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.167+0000 7f1ffa7a7700 1 -- start start 2023-10-06T01:41:14.173 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.168+0000 7f1ffa7a7700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1ff40ad840 0x7f1ff416a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:14.173 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.168+0000 7f1ffa7a7700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ff40adc10 0x7f1ff416ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:14.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.168+0000 7f1ffa7a7700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1ff416b370 0x7f1ff40a6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:14.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.168+0000 7f1ffa7a7700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff416abf0 con 0x7f1ff40adc10 2023-10-06T01:41:14.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.168+0000 7f1ffa7a7700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff40cb220 con 0x7f1ff416b370 2023-10-06T01:41:14.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.168+0000 7f1ffa7a7700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff40ce090 con 0x7f1ff40ad840 2023-10-06T01:41:14.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.168+0000 7f1ff37fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ff40adc10 0x7f1ff416ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:14.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.168+0000 7f1ff37fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ff40adc10 0x7f1ff416ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:40374/0 (socket says 172.21.15.27:40374) 2023-10-06T01:41:14.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.168+0000 7f1ff37fe700 1 -- 172.21.15.27:0/4094838828 learned_addr learned my addr 172.21.15.27:0/4094838828 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:14.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.168+0000 7f1ff3fff700 1 --2- 172.21.15.27:0/4094838828 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1ff40ad840 0x7f1ff416a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:14.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.168+0000 7f1ff8d44700 1 --2- 172.21.15.27:0/4094838828 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1ff416b370 0x7f1ff40a6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:14.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.169+0000 7f1ff37fe700 1 -- 172.21.15.27:0/4094838828 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1ff40ad840 msgr2=0x7f1ff416a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:14.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.169+0000 7f1ff37fe700 1 --2- 172.21.15.27:0/4094838828 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1ff40ad840 0x7f1ff416a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:14.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.169+0000 7f1ff37fe700 1 -- 172.21.15.27:0/4094838828 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1ff416b370 msgr2=0x7f1ff40a6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:14.175 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.169+0000 7f1ff37fe700 1 --2- 172.21.15.27:0/4094838828 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1ff416b370 0x7f1ff40a6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:14.175 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.169+0000 7f1ff37fe700 1 -- 172.21.15.27:0/4094838828 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff40ca610 con 0x7f1ff40adc10 2023-10-06T01:41:14.175 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.169+0000 7f1ff8d44700 1 --2- 172.21.15.27:0/4094838828 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1ff416b370 0x7f1ff40a6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:14.175 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.169+0000 7f1ff37fe700 1 --2- 172.21.15.27:0/4094838828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ff40adc10 0x7f1ff416ae30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1fe4011d50 tx=0x7f1fe40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:14.175 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.169+0000 7f1fdbfff700 1 -- 172.21.15.27:0/4094838828 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe400fac0 con 0x7f1ff40adc10 2023-10-06T01:41:14.175 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.169+0000 7f1fdbfff700 1 -- 172.21.15.27:0/4094838828 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fe400fc20 con 0x7f1ff40adc10 2023-10-06T01:41:14.175 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.169+0000 7f1ffa7a7700 1 -- 172.21.15.27:0/4094838828 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ff40ca8f0 con 0x7f1ff40adc10 2023-10-06T01:41:14.175 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.169+0000 7f1fdbfff700 1 -- 172.21.15.27:0/4094838828 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe401d480 con 0x7f1ff40adc10 2023-10-06T01:41:14.175 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.169+0000 7f1ffa7a7700 1 -- 172.21.15.27:0/4094838828 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ff40caeb0 con 0x7f1ff40adc10 2023-10-06T01:41:14.175 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.170+0000 7f1ffa7a7700 1 -- 172.21.15.27:0/4094838828 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ff4064050 con 0x7f1ff40adc10 2023-10-06T01:41:14.175 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.170+0000 7f1fdbfff700 1 -- 172.21.15.27:0/4094838828 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1fe401d8b0 con 0x7f1ff40adc10 2023-10-06T01:41:14.175 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.171+0000 7f1fdbfff700 1 --2- 172.21.15.27:0/4094838828 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1fe0066f30 0x7f1fe00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:14.176 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.171+0000 7f1fdbfff700 1 -- 172.21.15.27:0/4094838828 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f1fe4018070 con 0x7f1ff40adc10 2023-10-06T01:41:14.176 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.171+0000 7f1ff3fff700 1 --2- 172.21.15.27:0/4094838828 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1fe0066f30 0x7f1fe00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:14.176 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.171+0000 7f1ff3fff700 1 --2- 172.21.15.27:0/4094838828 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1fe0066f30 0x7f1fe00693e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1fe80079e0 tx=0x7f1ff40b2f40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:14.176 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.175+0000 7f1fdbfff700 1 -- 172.21.15.27:0/4094838828 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1fe4052590 con 0x7f1ff40adc10 2023-10-06T01:41:14.418 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:14.418+0000 7f1ffa7a7700 1 -- 172.21.15.27:0/4094838828 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f1ff4069290 con 0x7f1ff40adc10 2023-10-06T01:41:15.356 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.356+0000 7f1fdbfff700 1 -- 172.21.15.27:0/4094838828 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v6) v1 ==== 118+0+0 (secure 0 0 0) 0x7f1fe4055bb0 con 0x7f1ff40adc10 2023-10-06T01:41:15.359 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.359+0000 7f1ffa7a7700 1 -- 172.21.15.27:0/4094838828 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1fe0066f30 msgr2=0x7f1fe00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.359+0000 7f1ffa7a7700 1 --2- 172.21.15.27:0/4094838828 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1fe0066f30 0x7f1fe00693e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1fe80079e0 tx=0x7f1ff40b2f40 comp rx=0 tx=0).stop 2023-10-06T01:41:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.359+0000 7f1ffa7a7700 1 -- 172.21.15.27:0/4094838828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ff40adc10 msgr2=0x7f1ff416ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.359+0000 7f1ffa7a7700 1 --2- 172.21.15.27:0/4094838828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ff40adc10 0x7f1ff416ae30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f1fe4011d50 tx=0x7f1fe40091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.359+0000 7f1ffa7a7700 1 -- 172.21.15.27:0/4094838828 shutdown_connections 2023-10-06T01:41:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.359+0000 7f1ffa7a7700 1 --2- 172.21.15.27:0/4094838828 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1fe0066f30 0x7f1fe00693e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.359+0000 7f1ffa7a7700 1 --2- 172.21.15.27:0/4094838828 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1ff40ad840 0x7f1ff416a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.359+0000 7f1ffa7a7700 1 --2- 172.21.15.27:0/4094838828 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1ff416b370 0x7f1ff40a6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.359+0000 7f1ffa7a7700 1 --2- 172.21.15.27:0/4094838828 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1ff40adc10 0x7f1ff416ae30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.359+0000 7f1ffa7a7700 1 -- 172.21.15.27:0/4094838828 >> 172.21.15.27:0/4094838828 conn(0x7f1ff406d890 msgr2=0x7f1ff406c920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.359+0000 7f1ffa7a7700 1 -- 172.21.15.27:0/4094838828 shutdown_connections 2023-10-06T01:41:15.360 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.359+0000 7f1ffa7a7700 1 -- 172.21.15.27:0/4094838828 wait complete. 2023-10-06T01:41:15.374 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-06T01:41:15.503 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.503+0000 7fb88780a700 1 Processor -- start 2023-10-06T01:41:15.503 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.503+0000 7fb88780a700 1 -- start start 2023-10-06T01:41:15.505 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.504+0000 7fb88780a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb88015cb10 0x7fb8800af780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:15.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.504+0000 7fb88780a700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fb8800b4000 0x7fb880159b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:15.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.504+0000 7fb88780a700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8800698e0 con 0x7fb8800b1690 2023-10-06T01:41:15.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.504+0000 7fb88780a700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb880068fd0 con 0x7fb88015cb10 2023-10-06T01:41:15.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.504+0000 7fb88780a700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8800b0020 con 0x7fb8800b4000 2023-10-06T01:41:15.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.504+0000 7fb8855a6700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb88015cb10 0x7fb8800af780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:15.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.504+0000 7fb8855a6700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb88015cb10 0x7fb8800af780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55704/0 (socket says 172.21.15.27:55704) 2023-10-06T01:41:15.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.504+0000 7fb8855a6700 1 -- 172.21.15.27:0/1523213400 learned_addr learned my addr 172.21.15.27:0/1523213400 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:15.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.504+0000 7fb884da5700 1 --2- 172.21.15.27:0/1523213400 >> v2:172.21.15.124:3300/0 conn(0x7fb8800b4000 0x7fb880159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:15.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.504+0000 7fb8855a6700 1 -- 172.21.15.27:0/1523213400 >> v2:172.21.15.124:3300/0 conn(0x7fb8800b4000 msgr2=0x7fb880159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:15.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.504+0000 7fb8855a6700 1 --2- 172.21.15.27:0/1523213400 >> v2:172.21.15.124:3300/0 conn(0x7fb8800b4000 0x7fb880159b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:15.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.504+0000 7fb8855a6700 1 -- 172.21.15.27:0/1523213400 >> v1:172.21.15.124:6789/0 conn(0x7fb8800b1690 legacy=0x7fb880069d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:15.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.504+0000 7fb8855a6700 1 -- 172.21.15.27:0/1523213400 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb880069290 con 0x7fb88015cb10 2023-10-06T01:41:15.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.504+0000 7fb8855a6700 1 --2- 172.21.15.27:0/1523213400 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb88015cb10 0x7fb8800af780 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb88006f7c0 tx=0x7fb878009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=525aa463305388b0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:15.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.504+0000 7fb87ffff700 1 -- 172.21.15.27:0/1523213400 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87800b860 con 0x7fb88015cb10 2023-10-06T01:41:15.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.505+0000 7fb87ffff700 1 -- 172.21.15.27:0/1523213400 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb87800b9c0 con 0x7fb88015cb10 2023-10-06T01:41:15.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.505+0000 7fb87ffff700 1 -- 172.21.15.27:0/1523213400 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb87800bd60 con 0x7fb88015cb10 2023-10-06T01:41:15.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.505+0000 7fb88780a700 1 -- 172.21.15.27:0/1523213400 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb88015cb10 msgr2=0x7fb8800af780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:15.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.505+0000 7fb88780a700 1 --2- 172.21.15.27:0/1523213400 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb88015cb10 0x7fb8800af780 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb88006f7c0 tx=0x7fb878009250 comp rx=0 tx=0).stop 2023-10-06T01:41:15.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.505+0000 7fb88780a700 1 -- 172.21.15.27:0/1523213400 shutdown_connections 2023-10-06T01:41:15.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.505+0000 7fb88780a700 1 --2- 172.21.15.27:0/1523213400 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb88015cb10 0x7fb8800af780 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:15.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.505+0000 7fb88780a700 1 --2- 172.21.15.27:0/1523213400 >> v2:172.21.15.124:3300/0 conn(0x7fb8800b4000 0x7fb880159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:15.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.505+0000 7fb88780a700 1 -- 172.21.15.27:0/1523213400 >> 172.21.15.27:0/1523213400 conn(0x7fb88006d890 msgr2=0x7fb88006dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:15.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.505+0000 7fb88780a700 1 -- 172.21.15.27:0/1523213400 shutdown_connections 2023-10-06T01:41:15.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.505+0000 7fb88780a700 1 -- 172.21.15.27:0/1523213400 wait complete. 2023-10-06T01:41:15.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.506+0000 7fb88780a700 1 Processor -- start 2023-10-06T01:41:15.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.506+0000 7fb88780a700 1 -- start start 2023-10-06T01:41:15.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.507+0000 7fb88780a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8800b1690 0x7fb880147bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:15.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.507+0000 7fb88780a700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb8800b4000 0x7fb88016d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:15.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.507+0000 7fb88780a700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb88016d940 0x7fb8800a6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:15.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.507+0000 7fb88780a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8800a54c0 con 0x7fb8800b1690 2023-10-06T01:41:15.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.507+0000 7fb88780a700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb88016ddc0 con 0x7fb88016d940 2023-10-06T01:41:15.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.507+0000 7fb88780a700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8800a62b0 con 0x7fb8800b4000 2023-10-06T01:41:15.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.507+0000 7fb8855a6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8800b1690 0x7fb880147bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:15.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.507+0000 7fb8855a6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8800b1690 0x7fb880147bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55710/0 (socket says 172.21.15.27:55710) 2023-10-06T01:41:15.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.507+0000 7fb8855a6700 1 -- 172.21.15.27:0/3293155103 learned_addr learned my addr 172.21.15.27:0/3293155103 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:15.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.507+0000 7fb8855a6700 1 -- 172.21.15.27:0/3293155103 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb8800b4000 msgr2=0x7fb88016d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:15.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.507+0000 7fb8855a6700 1 --2- 172.21.15.27:0/3293155103 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb8800b4000 0x7fb88016d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:15.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.508+0000 7fb8855a6700 1 -- 172.21.15.27:0/3293155103 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb88016d940 msgr2=0x7fb8800a6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:15.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.508+0000 7fb8855a6700 1 --2- 172.21.15.27:0/3293155103 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb88016d940 0x7fb8800a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:15.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.508+0000 7fb8855a6700 1 -- 172.21.15.27:0/3293155103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8780094f0 con 0x7fb8800b1690 2023-10-06T01:41:15.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.508+0000 7fb8855a6700 1 --2- 172.21.15.27:0/3293155103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8800b1690 0x7fb880147bd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb88006acb0 tx=0x7fb878017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:15.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.508+0000 7fb87d7fa700 1 -- 172.21.15.27:0/3293155103 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb878024440 con 0x7fb8800b1690 2023-10-06T01:41:15.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.508+0000 7fb87d7fa700 1 -- 172.21.15.27:0/3293155103 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8780245a0 con 0x7fb8800b1690 2023-10-06T01:41:15.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.508+0000 7fb87d7fa700 1 -- 172.21.15.27:0/3293155103 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb878024960 con 0x7fb8800b1690 2023-10-06T01:41:15.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.508+0000 7fb88780a700 1 -- 172.21.15.27:0/3293155103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8800a56a0 con 0x7fb8800b1690 2023-10-06T01:41:15.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.508+0000 7fb88780a700 1 -- 172.21.15.27:0/3293155103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8800a5ce0 con 0x7fb8800b1690 2023-10-06T01:41:15.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.509+0000 7fb88780a700 1 -- 172.21.15.27:0/3293155103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb880064050 con 0x7fb8800b1690 2023-10-06T01:41:15.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.509+0000 7fb87d7fa700 1 -- 172.21.15.27:0/3293155103 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb87801a420 con 0x7fb8800b1690 2023-10-06T01:41:15.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.509+0000 7fb87d7fa700 1 --2- 172.21.15.27:0/3293155103 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb86c066f30 0x7fb86c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:15.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.510+0000 7fb87d7fa700 1 -- 172.21.15.27:0/3293155103 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fb8780862f0 con 0x7fb8800b1690 2023-10-06T01:41:15.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.510+0000 7fb884da5700 1 --2- 172.21.15.27:0/3293155103 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb86c066f30 0x7fb86c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:15.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.510+0000 7fb884da5700 1 --2- 172.21.15.27:0/3293155103 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb86c066f30 0x7fb86c0693e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb8800b09d0 tx=0x7fb874008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:15.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.514+0000 7fb87d7fa700 1 -- 172.21.15.27:0/3293155103 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb878053400 con 0x7fb8800b1690 2023-10-06T01:41:15.752 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:15.752+0000 7fb88780a700 1 -- 172.21.15.27:0/3293155103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb880069570 con 0x7fb8800b1690 2023-10-06T01:41:16.363 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.363+0000 7fb87d7fa700 1 -- 172.21.15.27:0/3293155103 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v7) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb878056a20 con 0x7fb8800b1690 2023-10-06T01:41:16.366 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.365+0000 7fb88780a700 1 -- 172.21.15.27:0/3293155103 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb86c066f30 msgr2=0x7fb86c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:16.366 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.365+0000 7fb88780a700 1 --2- 172.21.15.27:0/3293155103 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb86c066f30 0x7fb86c0693e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb8800b09d0 tx=0x7fb874008040 comp rx=0 tx=0).stop 2023-10-06T01:41:16.366 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.365+0000 7fb88780a700 1 -- 172.21.15.27:0/3293155103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8800b1690 msgr2=0x7fb880147bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:16.366 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.365+0000 7fb88780a700 1 --2- 172.21.15.27:0/3293155103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8800b1690 0x7fb880147bd0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb88006acb0 tx=0x7fb878017000 comp rx=0 tx=0).stop 2023-10-06T01:41:16.366 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.365+0000 7fb88780a700 1 -- 172.21.15.27:0/3293155103 shutdown_connections 2023-10-06T01:41:16.366 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.365+0000 7fb88780a700 1 --2- 172.21.15.27:0/3293155103 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb86c066f30 0x7fb86c0693e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:16.366 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.365+0000 7fb88780a700 1 --2- 172.21.15.27:0/3293155103 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb88016d940 0x7fb8800a6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:16.366 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.365+0000 7fb88780a700 1 --2- 172.21.15.27:0/3293155103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb8800b1690 0x7fb880147bd0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:16.367 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.365+0000 7fb88780a700 1 --2- 172.21.15.27:0/3293155103 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb8800b4000 0x7fb88016d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:16.367 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.365+0000 7fb88780a700 1 -- 172.21.15.27:0/3293155103 >> 172.21.15.27:0/3293155103 conn(0x7fb88006d890 msgr2=0x7fb88015a440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:16.367 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.366+0000 7fb88780a700 1 -- 172.21.15.27:0/3293155103 shutdown_connections 2023-10-06T01:41:16.367 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.366+0000 7fb88780a700 1 -- 172.21.15.27:0/3293155103 wait complete. 2023-10-06T01:41:16.379 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T01:41:16.499 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.498+0000 7fadf7e5e700 1 Processor -- start 2023-10-06T01:41:16.499 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.499+0000 7fadf7e5e700 1 -- start start 2023-10-06T01:41:16.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.499+0000 7fadf7e5e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fadf015d560 0x7fadf00d0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:16.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.499+0000 7fadf7e5e700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fadf01591b0 0x7fadf0159630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:16.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.499+0000 7fadf7e5e700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadf00b2ad0 con 0x7fadf015d930 2023-10-06T01:41:16.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.500+0000 7fadf7e5e700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadf00b4160 con 0x7fadf01591b0 2023-10-06T01:41:16.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.500+0000 7fadf7e5e700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadf0159b70 con 0x7fadf015d560 2023-10-06T01:41:16.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.500+0000 7fadf53f9700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fadf015d930 0x7fadf00b1a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46162/0 (socket says 172.21.15.27:46162) 2023-10-06T01:41:16.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.500+0000 7fadf53f9700 1 -- 172.21.15.27:0/1114116615 learned_addr learned my addr 172.21.15.27:0/1114116615 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:16.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.500+0000 7fadf5bfa700 1 --2- 172.21.15.27:0/1114116615 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fadf015d560 0x7fadf00d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:16.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.500+0000 7fadf63fb700 1 --2- 172.21.15.27:0/1114116615 >> v2:172.21.15.124:3300/0 conn(0x7fadf01591b0 0x7fadf0159630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:16.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.500+0000 7fadf4bf8700 1 -- 172.21.15.27:0/1114116615 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 527843369 0 0) 0x7fadf00b2ad0 con 0x7fadf015d930 2023-10-06T01:41:16.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.500+0000 7fadf4bf8700 1 -- 172.21.15.27:0/1114116615 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadd8003570 con 0x7fadf015d930 2023-10-06T01:41:16.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.500+0000 7fadf4bf8700 1 -- 172.21.15.27:0/1114116615 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 21250010 0 0) 0x7fadd8003570 con 0x7fadf015d930 2023-10-06T01:41:16.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.500+0000 7fadf63fb700 1 -- 172.21.15.27:0/1114116615 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fadf015d560 msgr2=0x7fadf00d0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:16.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.500+0000 7fadf63fb700 1 --2- 172.21.15.27:0/1114116615 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fadf015d560 0x7fadf00d0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:16.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.500+0000 7fadf63fb700 1 -- 172.21.15.27:0/1114116615 >> v1:172.21.15.27:6789/0 conn(0x7fadf015d930 legacy=0x7fadf00b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:16.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.500+0000 7fadf63fb700 1 -- 172.21.15.27:0/1114116615 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadf00b2230 con 0x7fadf01591b0 2023-10-06T01:41:16.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.500+0000 7fadf5bfa700 1 --2- 172.21.15.27:0/1114116615 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fadf015d560 0x7fadf00d0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:41:16.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.500+0000 7fadf63fb700 1 --2- 172.21.15.27:0/1114116615 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fadf01591b0 0x7fadf0159630 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fade0009e60 tx=0x7fade0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e479127eff7a1234 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:16.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.501+0000 7fadf4bf8700 1 -- 172.21.15.27:0/1114116615 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade000e070 con 0x7fadf01591b0 2023-10-06T01:41:16.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.501+0000 7fadf4bf8700 1 -- 172.21.15.27:0/1114116615 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fade000b7a0 con 0x7fadf01591b0 2023-10-06T01:41:16.503 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.501+0000 7fadf4bf8700 1 -- 172.21.15.27:0/1114116615 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade4001470 con 0x7fadf01591b0 2023-10-06T01:41:16.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.501+0000 7fadf7e5e700 1 -- 172.21.15.27:0/1114116615 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fadf01591b0 msgr2=0x7fadf0159630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:16.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.501+0000 7fadf7e5e700 1 --2- 172.21.15.27:0/1114116615 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fadf01591b0 0x7fadf0159630 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fade0009e60 tx=0x7fade0009780 comp rx=0 tx=0).stop 2023-10-06T01:41:16.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.501+0000 7fadf7e5e700 1 -- 172.21.15.27:0/1114116615 shutdown_connections 2023-10-06T01:41:16.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.501+0000 7fadf7e5e700 1 --2- 172.21.15.27:0/1114116615 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fadf01591b0 0x7fadf0159630 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:16.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.501+0000 7fadf7e5e700 1 --2- 172.21.15.27:0/1114116615 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fadf015d560 0x7fadf00d0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:16.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.501+0000 7fadf7e5e700 1 -- 172.21.15.27:0/1114116615 >> 172.21.15.27:0/1114116615 conn(0x7fadf00d1180 msgr2=0x7fadf0069640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:16.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.501+0000 7fadf7e5e700 1 -- 172.21.15.27:0/1114116615 shutdown_connections 2023-10-06T01:41:16.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.501+0000 7fadf7e5e700 1 -- 172.21.15.27:0/1114116615 wait complete. 2023-10-06T01:41:16.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.502+0000 7fadf7e5e700 1 Processor -- start 2023-10-06T01:41:16.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.502+0000 7fadf7e5e700 1 -- start start 2023-10-06T01:41:16.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.503+0000 7fadf7e5e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadf01591b0 0x7fadf00b10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:16.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.503+0000 7fadf7e5e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fadf015d560 0x7fadf0143100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:16.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.503+0000 7fadf7e5e700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fadf015d930 0x7fadf0155780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:16.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.503+0000 7fadf7e5e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf015c070 con 0x7fadf01591b0 2023-10-06T01:41:16.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.503+0000 7fadf7e5e700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadf0143640 con 0x7fadf015d930 2023-10-06T01:41:16.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.503+0000 7fadf7e5e700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadf015d010 con 0x7fadf015d560 2023-10-06T01:41:16.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.503+0000 7fadf5bfa700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadf01591b0 0x7fadf00b10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:16.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.503+0000 7fadf5bfa700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadf01591b0 0x7fadf00b10e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55724/0 (socket says 172.21.15.27:55724) 2023-10-06T01:41:16.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.503+0000 7fadf5bfa700 1 -- 172.21.15.27:0/1230720363 learned_addr learned my addr 172.21.15.27:0/1230720363 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:16.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.503+0000 7fadf53f9700 1 --2- 172.21.15.27:0/1230720363 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fadf015d560 0x7fadf0143100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:16.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.503+0000 7fadf63fb700 1 --2- 172.21.15.27:0/1230720363 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fadf015d930 0x7fadf0155780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:16.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.504+0000 7fadf5bfa700 1 -- 172.21.15.27:0/1230720363 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fadf015d560 msgr2=0x7fadf0143100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:16.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.504+0000 7fadf5bfa700 1 --2- 172.21.15.27:0/1230720363 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fadf015d560 0x7fadf0143100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:16.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.504+0000 7fadf5bfa700 1 -- 172.21.15.27:0/1230720363 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fadf015d930 msgr2=0x7fadf0155780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:16.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.504+0000 7fadf5bfa700 1 --2- 172.21.15.27:0/1230720363 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fadf015d930 0x7fadf0155780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:16.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.504+0000 7fadf5bfa700 1 -- 172.21.15.27:0/1230720363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade8009420 con 0x7fadf01591b0 2023-10-06T01:41:16.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.504+0000 7fadf5bfa700 1 --2- 172.21.15.27:0/1230720363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadf01591b0 0x7fadf00b10e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fade800ecb0 tx=0x7fade80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:16.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.504+0000 7fadedffb700 1 -- 172.21.15.27:0/1230720363 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade800c920 con 0x7fadf01591b0 2023-10-06T01:41:16.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.504+0000 7fadedffb700 1 -- 172.21.15.27:0/1230720363 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fade800ca80 con 0x7fadf01591b0 2023-10-06T01:41:16.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.504+0000 7fadf7e5e700 1 -- 172.21.15.27:0/1230720363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fade0009a20 con 0x7fadf01591b0 2023-10-06T01:41:16.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.504+0000 7fadedffb700 1 -- 172.21.15.27:0/1230720363 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fade800ce00 con 0x7fadf01591b0 2023-10-06T01:41:16.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.504+0000 7fadf7e5e700 1 -- 172.21.15.27:0/1230720363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadf0150e50 con 0x7fadf01591b0 2023-10-06T01:41:16.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.505+0000 7fadf7e5e700 1 -- 172.21.15.27:0/1230720363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadf015c350 con 0x7fadf01591b0 2023-10-06T01:41:16.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.505+0000 7fadedffb700 1 -- 172.21.15.27:0/1230720363 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fade8013680 con 0x7fadf01591b0 2023-10-06T01:41:16.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.506+0000 7fadedffb700 1 --2- 172.21.15.27:0/1230720363 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7faddc066f30 0x7faddc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:16.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.506+0000 7fadedffb700 1 -- 172.21.15.27:0/1230720363 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fade8084930 con 0x7fadf01591b0 2023-10-06T01:41:16.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.506+0000 7fadf53f9700 1 --2- 172.21.15.27:0/1230720363 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7faddc066f30 0x7faddc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:16.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.507+0000 7fadf53f9700 1 --2- 172.21.15.27:0/1230720363 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7faddc066f30 0x7faddc0693e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fadf015e300 tx=0x7fade4000bc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:16.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.510+0000 7fadedffb700 1 -- 172.21.15.27:0/1230720363 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fade8051a40 con 0x7fadf01591b0 2023-10-06T01:41:16.748 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.748+0000 7fadf7e5e700 1 -- 172.21.15.27:0/1230720363 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fadf0064050 con 0x7fadf01591b0 2023-10-06T01:41:16.749 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.749+0000 7fadedffb700 1 -- 172.21.15.27:0/1230720363 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4594 (secure 0 0 0) 0x7fade8055060 con 0x7fadf01591b0 2023-10-06T01:41:16.749 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:16.749 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4261,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6837/2699906167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6836","nonce":2699906167},{"type":"v1","addr":"172.21.15.124:6837","nonce":2699906167}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/313372271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":313372271},{"type":"v1","addr":"172.21.15.27:6834","nonce":313372271}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4287,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/4257743842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4257743842},{"type":"v1","addr":"172.21.15.27:6835","nonce":4257743842}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T01:41:12.735970+0000","modified":"2023-10-06T01:41:16.358924+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.124:6835/3803994994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":3803994994},{"type":"v1","addr":"172.21.15.124:6835","nonce":3803994994}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-06T01:41:16.751 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.751+0000 7fadf7e5e700 1 -- 172.21.15.27:0/1230720363 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7faddc066f30 msgr2=0x7faddc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:16.752 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.751+0000 7fadf7e5e700 1 --2- 172.21.15.27:0/1230720363 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7faddc066f30 0x7faddc0693e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fadf015e300 tx=0x7fade4000bc0 comp rx=0 tx=0).stop 2023-10-06T01:41:16.752 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.751+0000 7fadf7e5e700 1 -- 172.21.15.27:0/1230720363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadf01591b0 msgr2=0x7fadf00b10e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:16.752 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.751+0000 7fadf7e5e700 1 --2- 172.21.15.27:0/1230720363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadf01591b0 0x7fadf00b10e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fade800ecb0 tx=0x7fade80091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:16.752 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.751+0000 7fadf7e5e700 1 -- 172.21.15.27:0/1230720363 shutdown_connections 2023-10-06T01:41:16.752 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.751+0000 7fadf7e5e700 1 --2- 172.21.15.27:0/1230720363 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7faddc066f30 0x7faddc0693e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:16.752 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.751+0000 7fadf7e5e700 1 --2- 172.21.15.27:0/1230720363 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fadf015d930 0x7fadf0155780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:16.752 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.751+0000 7fadf7e5e700 1 --2- 172.21.15.27:0/1230720363 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fadf01591b0 0x7fadf00b10e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:16.752 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.751+0000 7fadf7e5e700 1 --2- 172.21.15.27:0/1230720363 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fadf015d560 0x7fadf0143100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:16.752 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.751+0000 7fadf7e5e700 1 -- 172.21.15.27:0/1230720363 >> 172.21.15.27:0/1230720363 conn(0x7fadf00d1180 msgr2=0x7fadf00b5480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:16.752 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.751+0000 7fadf7e5e700 1 -- 172.21.15.27:0/1230720363 shutdown_connections 2023-10-06T01:41:16.752 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.751+0000 7fadf7e5e700 1 -- 172.21.15.27:0/1230720363 wait complete. 2023-10-06T01:41:16.753 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 7 2023-10-06T01:41:16.763 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T01:41:16.881 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.881+0000 7f7d19081700 1 Processor -- start 2023-10-06T01:41:16.881 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.881+0000 7f7d19081700 1 -- start start 2023-10-06T01:41:16.884 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.882+0000 7f7d19081700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f7d140b5600 0x7f7d140d0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:16.884 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.882+0000 7f7d19081700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d140b2ad0 con 0x7f7d1415d930 2023-10-06T01:41:16.884 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.882+0000 7f7d19081700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d140b4160 con 0x7f7d1415d560 2023-10-06T01:41:16.885 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.882+0000 7f7d19081700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d140b2230 con 0x7f7d140b5600 2023-10-06T01:41:16.885 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.882+0000 7f7d1259c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7d1415d930 0x7f7d141591b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46178/0 (socket says 172.21.15.27:46178) 2023-10-06T01:41:16.885 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.882+0000 7f7d1259c700 1 -- 172.21.15.27:0/1326256885 learned_addr learned my addr 172.21.15.27:0/1326256885 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:16.885 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.882+0000 7f7d11d9b700 1 -- 172.21.15.27:0/1326256885 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3765839394 0 0) 0x7f7d140b2ad0 con 0x7f7d1415d930 2023-10-06T01:41:16.885 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.882+0000 7f7d1359e700 1 --2- 172.21.15.27:0/1326256885 >> v2:172.21.15.124:3300/0 conn(0x7f7d140b5600 0x7f7d140d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:16.885 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.882+0000 7f7d11d9b700 1 -- 172.21.15.27:0/1326256885 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d00003570 con 0x7f7d1415d930 2023-10-06T01:41:16.885 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.882+0000 7f7d11d9b700 1 -- 172.21.15.27:0/1326256885 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3980857262 0 0) 0x7f7d00003570 con 0x7f7d1415d930 2023-10-06T01:41:16.885 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.882+0000 7f7d11d9b700 1 -- 172.21.15.27:0/1326256885 >> v2:172.21.15.124:3300/0 conn(0x7f7d140b5600 msgr2=0x7f7d140d0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:16.885 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.882+0000 7f7d11d9b700 1 --2- 172.21.15.27:0/1326256885 >> v2:172.21.15.124:3300/0 conn(0x7f7d140b5600 0x7f7d140d0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:16.885 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.882+0000 7f7d11d9b700 1 -- 172.21.15.27:0/1326256885 >> v1:172.21.15.124:6789/0 conn(0x7f7d1415d560 legacy=0x7f7d140b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:16.885 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.882+0000 7f7d11d9b700 1 -- 172.21.15.27:0/1326256885 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d140b2510 con 0x7f7d1415d930 2023-10-06T01:41:16.886 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.882+0000 7f7d11d9b700 1 -- 172.21.15.27:0/1326256885 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2619078944 0 0) 0x7f7cfc001470 con 0x7f7d1415d930 2023-10-06T01:41:16.886 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.882+0000 7f7d11d9b700 1 -- 172.21.15.27:0/1326256885 >> v1:172.21.15.27:6789/0 conn(0x7f7d1415d930 legacy=0x7f7d141591b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:16.886 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.882+0000 7f7d11d9b700 1 --2- 172.21.15.27:0/1326256885 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7d00003a60 0x7f7d00003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:16.886 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.882+0000 7f7d11d9b700 1 --2- 172.21.15.27:0/1326256885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d00004810 0x7f7d00004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:16.886 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.882+0000 7f7d11d9b700 1 --2- 172.21.15.27:0/1326256885 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7d000051c0 0x7f7d0000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:16.886 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.882+0000 7f7d11d9b700 1 -- 172.21.15.27:0/1326256885 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d140b2230 con 0x7f7d00004810 2023-10-06T01:41:16.886 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.882+0000 7f7d11d9b700 1 -- 172.21.15.27:0/1326256885 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0000c160 con 0x7f7d000051c0 2023-10-06T01:41:16.886 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.882+0000 7f7d11d9b700 1 -- 172.21.15.27:0/1326256885 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0000c320 con 0x7f7d00003a60 2023-10-06T01:41:16.886 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.882+0000 7f7d1259c700 1 --2- 172.21.15.27:0/1326256885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d00004810 0x7f7d00004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:16.886 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.882+0000 7f7d12d9d700 1 --2- 172.21.15.27:0/1326256885 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7d00003a60 0x7f7d00003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:16.886 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.882+0000 7f7d1359e700 1 --2- 172.21.15.27:0/1326256885 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7d000051c0 0x7f7d0000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:16.886 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.883+0000 7f7d1259c700 1 -- 172.21.15.27:0/1326256885 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7d00003a60 msgr2=0x7f7d00003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:16.886 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.883+0000 7f7d1259c700 1 --2- 172.21.15.27:0/1326256885 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7d00003a60 0x7f7d00003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:16.887 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.883+0000 7f7d1259c700 1 -- 172.21.15.27:0/1326256885 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7d000051c0 msgr2=0x7f7d0000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:16.887 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.883+0000 7f7d1259c700 1 --2- 172.21.15.27:0/1326256885 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7d000051c0 0x7f7d0000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:16.887 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.883+0000 7f7d1259c700 1 -- 172.21.15.27:0/1326256885 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d00003570 con 0x7f7d00004810 2023-10-06T01:41:16.887 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.883+0000 7f7d1359e700 1 -- 172.21.15.27:0/1326256885 reap_dead start 2023-10-06T01:41:16.887 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.883+0000 7f7d1259c700 1 --2- 172.21.15.27:0/1326256885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d00004810 0x7f7d00004c80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc00c3c0 tx=0x7f7cfc001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d73a89972b711cc server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:16.887 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.883+0000 7f7d11d9b700 1 -- 172.21.15.27:0/1326256885 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cfc00cc10 con 0x7f7d00004810 2023-10-06T01:41:16.887 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.883+0000 7f7d11d9b700 1 -- 172.21.15.27:0/1326256885 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cfc00cd70 con 0x7f7d00004810 2023-10-06T01:41:16.887 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.883+0000 7f7d11d9b700 1 -- 172.21.15.27:0/1326256885 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cfc005b50 con 0x7f7d00004810 2023-10-06T01:41:16.887 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.883+0000 7f7d19081700 1 -- 172.21.15.27:0/1326256885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d00004810 msgr2=0x7f7d00004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:16.887 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.883+0000 7f7d19081700 1 --2- 172.21.15.27:0/1326256885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d00004810 0x7f7d00004c80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc00c3c0 tx=0x7f7cfc001e50 comp rx=0 tx=0).stop 2023-10-06T01:41:16.887 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.884+0000 7f7d19081700 1 -- 172.21.15.27:0/1326256885 shutdown_connections 2023-10-06T01:41:16.887 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.884+0000 7f7d19081700 1 --2- 172.21.15.27:0/1326256885 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d00004810 0x7f7d00004c80 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:16.888 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.884+0000 7f7d19081700 1 -- 172.21.15.27:0/1326256885 >> 172.21.15.27:0/1326256885 conn(0x7f7d140d1180 msgr2=0x7f7d14069640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:16.888 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.884+0000 7f7d19081700 1 -- 172.21.15.27:0/1326256885 shutdown_connections 2023-10-06T01:41:16.888 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.884+0000 7f7d19081700 1 -- 172.21.15.27:0/1326256885 wait complete. 2023-10-06T01:41:16.890 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.885+0000 7f7d19081700 1 Processor -- start 2023-10-06T01:41:16.890 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.885+0000 7f7d19081700 1 -- start start 2023-10-06T01:41:16.891 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.885+0000 7f7d19081700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d00004810 0x7f7d14156270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:16.891 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.886+0000 7f7d19081700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7d14159950 0x7f7d1416d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:16.891 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.886+0000 7f7d19081700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7d14159dc0 0x7f7d1416da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:16.891 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.886+0000 7f7d19081700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1415bf30 con 0x7f7d00004810 2023-10-06T01:41:16.891 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.886+0000 7f7d19081700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1415cd70 con 0x7f7d14159dc0 2023-10-06T01:41:16.891 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.886+0000 7f7d19081700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1415c160 con 0x7f7d14159950 2023-10-06T01:41:16.891 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.886+0000 7f7d12d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d00004810 0x7f7d14156270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:16.891 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.886+0000 7f7d12d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d00004810 0x7f7d14156270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55750/0 (socket says 172.21.15.27:55750) 2023-10-06T01:41:16.891 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.886+0000 7f7d12d9d700 1 -- 172.21.15.27:0/2942426685 learned_addr learned my addr 172.21.15.27:0/2942426685 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:16.891 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.886+0000 7f7d1359e700 1 --2- 172.21.15.27:0/2942426685 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7d14159dc0 0x7f7d1416da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:16.891 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.886+0000 7f7d1259c700 1 --2- 172.21.15.27:0/2942426685 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7d14159950 0x7f7d1416d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:16.891 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.886+0000 7f7d12d9d700 1 -- 172.21.15.27:0/2942426685 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7d14159950 msgr2=0x7f7d1416d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:16.893 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.886+0000 7f7d12d9d700 1 --2- 172.21.15.27:0/2942426685 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7d14159950 0x7f7d1416d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:16.893 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.886+0000 7f7d12d9d700 1 -- 172.21.15.27:0/2942426685 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7d14159dc0 msgr2=0x7f7d1416da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:16.894 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.886+0000 7f7d12d9d700 1 --2- 172.21.15.27:0/2942426685 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7d14159dc0 0x7f7d1416da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:16.894 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.886+0000 7f7d12d9d700 1 -- 172.21.15.27:0/2942426685 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d1415c440 con 0x7f7d00004810 2023-10-06T01:41:16.894 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.886+0000 7f7d1359e700 1 --2- 172.21.15.27:0/2942426685 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7d14159dc0 0x7f7d1416da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:41:16.894 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.886+0000 7f7d1259c700 1 --2- 172.21.15.27:0/2942426685 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7d14159950 0x7f7d1416d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:41:16.894 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.886+0000 7f7d12d9d700 1 --2- 172.21.15.27:0/2942426685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d00004810 0x7f7d14156270 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7d0400eb10 tx=0x7f7d04001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:16.894 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.886+0000 7f7d0affd700 1 -- 172.21.15.27:0/2942426685 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0400c890 con 0x7f7d00004810 2023-10-06T01:41:16.894 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.886+0000 7f7d0affd700 1 -- 172.21.15.27:0/2942426685 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d0400c9f0 con 0x7f7d00004810 2023-10-06T01:41:16.894 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.886+0000 7f7d19081700 1 -- 172.21.15.27:0/2942426685 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d1415c720 con 0x7f7d00004810 2023-10-06T01:41:16.894 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.886+0000 7f7d0affd700 1 -- 172.21.15.27:0/2942426685 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d0400cd90 con 0x7f7d00004810 2023-10-06T01:41:16.895 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.886+0000 7f7d19081700 1 -- 172.21.15.27:0/2942426685 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d14154670 con 0x7f7d00004810 2023-10-06T01:41:16.895 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.887+0000 7f7d19081700 1 -- 172.21.15.27:0/2942426685 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d14152580 con 0x7f7d00004810 2023-10-06T01:41:16.895 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.888+0000 7f7d0affd700 1 -- 172.21.15.27:0/2942426685 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7d04014690 con 0x7f7d00004810 2023-10-06T01:41:16.895 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.888+0000 7f7d0affd700 1 --2- 172.21.15.27:0/2942426685 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7cf4066f30 0x7f7cf40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:16.895 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.888+0000 7f7d0affd700 1 -- 172.21.15.27:0/2942426685 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f7d04084c90 con 0x7f7d00004810 2023-10-06T01:41:16.895 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.888+0000 7f7d1259c700 1 --2- 172.21.15.27:0/2942426685 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7cf4066f30 0x7f7cf40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:16.895 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.889+0000 7f7d1259c700 1 --2- 172.21.15.27:0/2942426685 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7cf4066f30 0x7f7cf40693e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7d140cd930 tx=0x7f7cfc015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:16.895 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:16.892+0000 7f7d0affd700 1 -- 172.21.15.27:0/2942426685 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7d04051ed0 con 0x7f7d00004810 2023-10-06T01:41:17.087 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.087+0000 7f7d19081700 1 -- 172.21.15.27:0/2942426685 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d1415cbc0 con 0x7f7d00004810 2023-10-06T01:41:17.088 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.088+0000 7f7d0affd700 1 -- 172.21.15.27:0/2942426685 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16866 (secure 0 0 0) 0x7f7d040554f0 con 0x7f7d00004810 2023-10-06T01:41:17.089 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:17.089 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":20,"fsid":"20a7bff0-74f8-4f2f-b6c6-899e804bc965","created":"2023-10-06T01:40:54.474215+0000","modified":"2023-10-06T01:41:14.324546+0000","last_up_change":"2023-10-06T01:41:05.242847+0000","last_in_change":"2023-10-06T01:40:56.890960+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-10-06T01:41:05.685386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-10-06T01:41:06.092995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T01:41:10.434257+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T01:41:11.700262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b902c86d-800c-40c5-b152-b545e50d02c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":90644},{"type":"v1","addr":"172.21.15.27:6801","nonce":90644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":90644},{"type":"v1","addr":"172.21.15.27:6803","nonce":90644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":90644},{"type":"v1","addr":"172.21.15.27:6807","nonce":90644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":90644},{"type":"v1","addr":"172.21.15.27:6805","nonce":90644}]},"public_addr":"172.21.15.27:6801/90644","cluster_addr":"172.21.15.27:6803/90644","heartbeat_back_addr":"172.21.15.27:6807/90644","heartbeat_front_addr":"172.21.15.27:6805/90644","state":["exists","up"]},{"osd":1,"uuid":"5b4a70e5-b06e-48cd-bfc4-7962c5d05386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":90643},{"type":"v1","addr":"172.21.15.27:6809","nonce":90643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":90643},{"type":"v1","addr":"172.21.15.27:6811","nonce":90643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":90643},{"type":"v1","addr":"172.21.15.27:6815","nonce":90643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":90643},{"type":"v1","addr":"172.21.15.27:6813","nonce":90643}]},"public_addr":"172.21.15.27:6809/90643","cluster_addr":"172.21.15.27:6811/90643","heartbeat_back_addr":"172.21.15.27:6815/90643","heartbeat_front_addr":"172.21.15.27:6813/90643","state":["exists","up"]},{"osd":2,"uuid":"22d9de46-4073-4336-9122-5227e601eca0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":90645},{"type":"v1","addr":"172.21.15.27:6817","nonce":90645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":90645},{"type":"v1","addr":"172.21.15.27:6819","nonce":90645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":90645},{"type":"v1","addr":"172.21.15.27:6823","nonce":90645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":90645},{"type":"v1","addr":"172.21.15.27:6821","nonce":90645}]},"public_addr":"172.21.15.27:6817/90645","cluster_addr":"172.21.15.27:6819/90645","heartbeat_back_addr":"172.21.15.27:6823/90645","heartbeat_front_addr":"172.21.15.27:6821/90645","state":["exists","up"]},{"osd":3,"uuid":"438bb2d3-34e2-4b7c-9734-8360c1b67650","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":90668},{"type":"v1","addr":"172.21.15.27:6825","nonce":90668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":90668},{"type":"v1","addr":"172.21.15.27:6827","nonce":90668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":90668},{"type":"v1","addr":"172.21.15.27:6831","nonce":90668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":90668},{"type":"v1","addr":"172.21.15.27:6829","nonce":90668}]},"public_addr":"172.21.15.27:6825/90668","cluster_addr":"172.21.15.27:6827/90668","heartbeat_back_addr":"172.21.15.27:6831/90668","heartbeat_front_addr":"172.21.15.27:6829/90668","state":["exists","up"]},{"osd":4,"uuid":"e1956a8e-e0f8-4e7c-acbb-40037415a354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":90863},{"type":"v1","addr":"172.21.15.124:6801","nonce":90863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":90863},{"type":"v1","addr":"172.21.15.124:6803","nonce":90863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":90863},{"type":"v1","addr":"172.21.15.124:6807","nonce":90863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":90863},{"type":"v1","addr":"172.21.15.124:6805","nonce":90863}]},"public_addr":"172.21.15.124:6801/90863","cluster_addr":"172.21.15.124:6803/90863","heartbeat_back_addr":"172.21.15.124:6807/90863","heartbeat_front_addr":"172.21.15.124:6805/90863","state":["exists","up"]},{"osd":5,"uuid":"306c59d4-846d-4b73-b1c9-051bec9e9368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":90885},{"type":"v1","addr":"172.21.15.124:6817","nonce":90885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":90885},{"type":"v1","addr":"172.21.15.124:6819","nonce":90885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":90885},{"type":"v1","addr":"172.21.15.124:6823","nonce":90885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":90885},{"type":"v1","addr":"172.21.15.124:6821","nonce":90885}]},"public_addr":"172.21.15.124:6817/90885","cluster_addr":"172.21.15.124:6819/90885","heartbeat_back_addr":"172.21.15.124:6823/90885","heartbeat_front_addr":"172.21.15.124:6821/90885","state":["exists","up"]},{"osd":6,"uuid":"15463e53-ac8d-47d1-b4f6-c0d31f29e4a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":90864},{"type":"v1","addr":"172.21.15.124:6809","nonce":90864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":90864},{"type":"v1","addr":"172.21.15.124:6811","nonce":90864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":90864},{"type":"v1","addr":"172.21.15.124:6815","nonce":90864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":90864},{"type":"v1","addr":"172.21.15.124:6813","nonce":90864}]},"public_addr":"172.21.15.124:6809/90864","cluster_addr":"172.21.15.124:6811/90864","heartbeat_back_addr":"172.21.15.124:6815/90864","heartbeat_front_addr":"172.21.15.124:6813/90864","state":["exists","up"]},{"osd":7,"uuid":"ae463a90-a9d3-498f-86fa-47a35b3fe889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":90894},{"type":"v1","addr":"172.21.15.124:6825","nonce":90894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":90894},{"type":"v1","addr":"172.21.15.124:6827","nonce":90894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":90894},{"type":"v1","addr":"172.21.15.124:6831","nonce":90894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":90894},{"type":"v1","addr":"172.21.15.124:6829","nonce":90894}]},"public_addr":"172.21.15.124:6825/90894","cluster_addr":"172.21.15.124:6827/90894","heartbeat_back_addr":"172.21.15.124:6831/90894","heartbeat_front_addr":"172.21.15.124:6829/90894","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.827704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.839951+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.032230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.050605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.987908+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.031290+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.080707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.053692+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T01:41:17.091 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.090+0000 7f7d19081700 1 -- 172.21.15.27:0/2942426685 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7cf4066f30 msgr2=0x7f7cf40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:17.091 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.090+0000 7f7d19081700 1 --2- 172.21.15.27:0/2942426685 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7cf4066f30 0x7f7cf40693e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7d140cd930 tx=0x7f7cfc015000 comp rx=0 tx=0).stop 2023-10-06T01:41:17.091 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.090+0000 7f7d19081700 1 -- 172.21.15.27:0/2942426685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d00004810 msgr2=0x7f7d14156270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:17.091 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.090+0000 7f7d19081700 1 --2- 172.21.15.27:0/2942426685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d00004810 0x7f7d14156270 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7d0400eb10 tx=0x7f7d04001640 comp rx=0 tx=0).stop 2023-10-06T01:41:17.091 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.090+0000 7f7d19081700 1 -- 172.21.15.27:0/2942426685 shutdown_connections 2023-10-06T01:41:17.091 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.090+0000 7f7d19081700 1 --2- 172.21.15.27:0/2942426685 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7cf4066f30 0x7f7cf40693e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:17.091 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.090+0000 7f7d19081700 1 --2- 172.21.15.27:0/2942426685 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7d14159dc0 0x7f7d1416da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:17.091 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.090+0000 7f7d19081700 1 --2- 172.21.15.27:0/2942426685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d00004810 0x7f7d14156270 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:17.091 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.090+0000 7f7d19081700 1 --2- 172.21.15.27:0/2942426685 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7d14159950 0x7f7d1416d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:17.091 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.090+0000 7f7d19081700 1 -- 172.21.15.27:0/2942426685 >> 172.21.15.27:0/2942426685 conn(0x7f7d140d1180 msgr2=0x7f7d14068270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:17.092 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.090+0000 7f7d19081700 1 -- 172.21.15.27:0/2942426685 shutdown_connections 2023-10-06T01:41:17.092 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.090+0000 7f7d19081700 1 -- 172.21.15.27:0/2942426685 wait complete. 2023-10-06T01:41:17.103 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-06T01:41:17.103 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-06T01:41:17.221 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.221+0000 7fd75eee0700 1 Processor -- start 2023-10-06T01:41:17.221 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.221+0000 7fd75eee0700 1 -- start start 2023-10-06T01:41:17.223 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.222+0000 7fd75eee0700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fd7580b0310 0x7fd7580b0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:17.223 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.222+0000 7fd75eee0700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd758069f70 con 0x7fd75815d480 2023-10-06T01:41:17.223 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.222+0000 7fd75eee0700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7580b0c90 con 0x7fd75815d850 2023-10-06T01:41:17.223 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.222+0000 7fd75eee0700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7580687c0 con 0x7fd7580b0310 2023-10-06T01:41:17.224 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.222+0000 7fd75dede700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd75815d480 0x7fd758069860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46188/0 (socket says 172.21.15.27:46188) 2023-10-06T01:41:17.224 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.222+0000 7fd75dede700 1 -- 172.21.15.27:0/2494920324 learned_addr learned my addr 172.21.15.27:0/2494920324 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:17.224 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.222+0000 7fd75e6df700 1 --2- 172.21.15.27:0/2494920324 >> v2:172.21.15.124:3300/0 conn(0x7fd7580b0310 0x7fd7580b0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:17.224 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.222+0000 7fd75cedc700 1 -- 172.21.15.27:0/2494920324 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 314907094 0 0) 0x7fd758069f70 con 0x7fd75815d480 2023-10-06T01:41:17.224 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.223+0000 7fd75cedc700 1 -- 172.21.15.27:0/2494920324 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd740003570 con 0x7fd75815d480 2023-10-06T01:41:17.224 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.223+0000 7fd75e6df700 1 -- 172.21.15.27:0/2494920324 >> v1:172.21.15.124:6789/0 conn(0x7fd75815d850 legacy=0x7fd7580b5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:17.224 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.223+0000 7fd75e6df700 1 -- 172.21.15.27:0/2494920324 >> v1:172.21.15.27:6789/0 conn(0x7fd75815d480 legacy=0x7fd758069860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:17.224 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.223+0000 7fd75e6df700 1 -- 172.21.15.27:0/2494920324 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd758068aa0 con 0x7fd7580b0310 2023-10-06T01:41:17.224 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.223+0000 7fd75e6df700 1 --2- 172.21.15.27:0/2494920324 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd7580b0310 0x7fd7580b0750 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd748009e60 tx=0x7fd748009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64dd556deab369db server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:17.224 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.223+0000 7fd75cedc700 1 -- 172.21.15.27:0/2494920324 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74800e070 con 0x7fd7580b0310 2023-10-06T01:41:17.224 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.223+0000 7fd75cedc700 1 -- 172.21.15.27:0/2494920324 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd74800bb60 con 0x7fd7580b0310 2023-10-06T01:41:17.224 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.223+0000 7fd75cedc700 1 -- 172.21.15.27:0/2494920324 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd748012430 con 0x7fd7580b0310 2023-10-06T01:41:17.227 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.224+0000 7fd75eee0700 1 -- 172.21.15.27:0/2494920324 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd7580b0310 msgr2=0x7fd7580b0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:17.227 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.224+0000 7fd75eee0700 1 --2- 172.21.15.27:0/2494920324 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd7580b0310 0x7fd7580b0750 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd748009e60 tx=0x7fd748009780 comp rx=0 tx=0).stop 2023-10-06T01:41:17.227 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.224+0000 7fd75eee0700 1 -- 172.21.15.27:0/2494920324 shutdown_connections 2023-10-06T01:41:17.227 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.224+0000 7fd75eee0700 1 --2- 172.21.15.27:0/2494920324 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd7580b0310 0x7fd7580b0750 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:17.227 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.224+0000 7fd75eee0700 1 -- 172.21.15.27:0/2494920324 >> 172.21.15.27:0/2494920324 conn(0x7fd7580ccf80 msgr2=0x7fd7580cd380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:17.228 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.224+0000 7fd75eee0700 1 -- 172.21.15.27:0/2494920324 shutdown_connections 2023-10-06T01:41:17.228 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.224+0000 7fd75eee0700 1 -- 172.21.15.27:0/2494920324 wait complete. 2023-10-06T01:41:17.228 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.225+0000 7fd75eee0700 1 Processor -- start 2023-10-06T01:41:17.228 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.225+0000 7fd75eee0700 1 -- start start 2023-10-06T01:41:17.228 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.226+0000 7fd75eee0700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd7580b0310 0x7fd7580ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:17.228 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.226+0000 7fd75eee0700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd75815d480 0x7fd7580c82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:17.228 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.226+0000 7fd75eee0700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd75815d850 0x7fd758159150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:17.228 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.226+0000 7fd75eee0700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7580c8810 con 0x7fd75815d850 2023-10-06T01:41:17.228 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.226+0000 7fd75eee0700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd75816cda0 con 0x7fd7580b0310 2023-10-06T01:41:17.228 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.226+0000 7fd75eee0700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd75816d080 con 0x7fd75815d480 2023-10-06T01:41:17.228 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.226+0000 7fd75e6df700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd75815d850 0x7fd758159150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:17.228 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.226+0000 7fd75e6df700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd75815d850 0x7fd758159150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55752/0 (socket says 172.21.15.27:55752) 2023-10-06T01:41:17.229 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.226+0000 7fd75e6df700 1 -- 172.21.15.27:0/4198970805 learned_addr learned my addr 172.21.15.27:0/4198970805 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:17.229 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.226+0000 7fd75dede700 1 --2- 172.21.15.27:0/4198970805 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd7580b0310 0x7fd7580ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:17.229 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.226+0000 7fd75d6dd700 1 --2- 172.21.15.27:0/4198970805 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd75815d480 0x7fd7580c82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:17.229 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.226+0000 7fd75e6df700 1 -- 172.21.15.27:0/4198970805 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd75815d480 msgr2=0x7fd7580c82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:17.229 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.226+0000 7fd75e6df700 1 --2- 172.21.15.27:0/4198970805 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd75815d480 0x7fd7580c82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:17.229 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.226+0000 7fd75e6df700 1 -- 172.21.15.27:0/4198970805 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd7580b0310 msgr2=0x7fd7580ca4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:17.229 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.226+0000 7fd75e6df700 1 --2- 172.21.15.27:0/4198970805 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd7580b0310 0x7fd7580ca4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:17.229 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.226+0000 7fd75e6df700 1 -- 172.21.15.27:0/4198970805 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd748009a20 con 0x7fd75815d850 2023-10-06T01:41:17.229 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.226+0000 7fd75e6df700 1 --2- 172.21.15.27:0/4198970805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd75815d850 0x7fd758159150 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd758088ca0 tx=0x7fd748019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:17.229 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.226+0000 7fd755ffb700 1 -- 172.21.15.27:0/4198970805 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd748027470 con 0x7fd75815d850 2023-10-06T01:41:17.229 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.226+0000 7fd755ffb700 1 -- 172.21.15.27:0/4198970805 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7480275f0 con 0x7fd75815d850 2023-10-06T01:41:17.230 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.226+0000 7fd75eee0700 1 -- 172.21.15.27:0/4198970805 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd75816d360 con 0x7fd75815d850 2023-10-06T01:41:17.230 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.226+0000 7fd755ffb700 1 -- 172.21.15.27:0/4198970805 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7480279b0 con 0x7fd75815d850 2023-10-06T01:41:17.230 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.227+0000 7fd75eee0700 1 -- 172.21.15.27:0/4198970805 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd75816d920 con 0x7fd75815d850 2023-10-06T01:41:17.231 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.228+0000 7fd75eee0700 1 -- 172.21.15.27:0/4198970805 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd75804c3c0 con 0x7fd75815d850 2023-10-06T01:41:17.231 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.228+0000 7fd755ffb700 1 -- 172.21.15.27:0/4198970805 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd748024070 con 0x7fd75815d850 2023-10-06T01:41:17.231 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.228+0000 7fd755ffb700 1 --2- 172.21.15.27:0/4198970805 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd744066f60 0x7fd744069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:17.231 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.228+0000 7fd755ffb700 1 -- 172.21.15.27:0/4198970805 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fd748086fe0 con 0x7fd75815d850 2023-10-06T01:41:17.231 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.229+0000 7fd75dede700 1 --2- 172.21.15.27:0/4198970805 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd744066f60 0x7fd744069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:17.231 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.229+0000 7fd75dede700 1 --2- 172.21.15.27:0/4198970805 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd744066f60 0x7fd744069410 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd750007fd0 tx=0x7fd750009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:17.233 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.233+0000 7fd755ffb700 1 -- 172.21.15.27:0/4198970805 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd748054070 con 0x7fd75815d850 2023-10-06T01:41:17.423 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.422+0000 7fd75eee0700 1 -- 172.21.15.27:0/4198970805 --> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd758068d80 con 0x7fd744066f60 2023-10-06T01:41:17.431 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.431+0000 7fd755ffb700 1 -- 172.21.15.27:0/4198970805 <== mgr.4100 v2:172.21.15.124:6832/90278 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+278408 (secure 0 0 0) 0x7fd758068d80 con 0x7fd744066f60 2023-10-06T01:41:17.435 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:17.437 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.434+0000 7fd75eee0700 1 -- 172.21.15.27:0/4198970805 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd744066f60 msgr2=0x7fd744069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:17.437 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.434+0000 7fd75eee0700 1 --2- 172.21.15.27:0/4198970805 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd744066f60 0x7fd744069410 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd750007fd0 tx=0x7fd750009040 comp rx=0 tx=0).stop 2023-10-06T01:41:17.437 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.434+0000 7fd75eee0700 1 -- 172.21.15.27:0/4198970805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd75815d850 msgr2=0x7fd758159150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:17.437 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.434+0000 7fd75eee0700 1 --2- 172.21.15.27:0/4198970805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd75815d850 0x7fd758159150 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd758088ca0 tx=0x7fd748019000 comp rx=0 tx=0).stop 2023-10-06T01:41:17.437 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.434+0000 7fd75eee0700 1 -- 172.21.15.27:0/4198970805 shutdown_connections 2023-10-06T01:41:17.438 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.434+0000 7fd75eee0700 1 --2- 172.21.15.27:0/4198970805 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd744066f60 0x7fd744069410 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:17.438 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.434+0000 7fd75eee0700 1 --2- 172.21.15.27:0/4198970805 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd75815d850 0x7fd758159150 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:17.438 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.434+0000 7fd75eee0700 1 --2- 172.21.15.27:0/4198970805 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd7580b0310 0x7fd7580ca4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:17.438 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.434+0000 7fd75eee0700 1 --2- 172.21.15.27:0/4198970805 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd75815d480 0x7fd7580c82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:17.438 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.434+0000 7fd75eee0700 1 -- 172.21.15.27:0/4198970805 >> 172.21.15.27:0/4198970805 conn(0x7fd7580ccf80 msgr2=0x7fd7581413c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:17.438 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.435+0000 7fd75eee0700 1 -- 172.21.15.27:0/4198970805 shutdown_connections 2023-10-06T01:41:17.438 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.435+0000 7fd75eee0700 1 -- 172.21.15.27:0/4198970805 wait complete. 2023-10-06T01:41:17.438 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-06T01:41:17.447 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2023-10-06T01:41:16.038592+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590922,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":167,"num_write_kb":2417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":102,"acting":102,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":52,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2381356,"kb_used_data":2488,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752593364,"statfs":{"total":773094113280,"available":770655604736,"internally_reserved":0,"allocated":2547712,"data_stored":1623543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":554,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.722059"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.342002+0000","last_change":"2023-10-06T01:41:13.342002+0000","last_active":"2023-10-06T01:41:13.342002+0000","last_peered":"2023-10-06T01:41:13.342002+0000","last_clean":"2023-10-06T01:41:13.342002+0000","last_became_active":"2023-10-06T01:41:13.341403+0000","last_became_peered":"2023-10-06T01:41:13.341403+0000","last_unstale":"2023-10-06T01:41:13.342002+0000","last_undegraded":"2023-10-06T01:41:13.342002+0000","last_fullsized":"2023-10-06T01:41:13.342002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:34:55.375989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.343792+0000","last_change":"2023-10-06T01:41:13.343792+0000","last_active":"2023-10-06T01:41:13.343792+0000","last_peered":"2023-10-06T01:41:13.343792+0000","last_clean":"2023-10-06T01:41:13.343792+0000","last_became_active":"2023-10-06T01:41:13.343432+0000","last_became_peered":"2023-10-06T01:41:13.343432+0000","last_unstale":"2023-10-06T01:41:13.343792+0000","last_undegraded":"2023-10-06T01:41:13.343792+0000","last_fullsized":"2023-10-06T01:41:13.343792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:52:48.286318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.344490+0000","last_change":"2023-10-06T01:41:13.344490+0000","last_active":"2023-10-06T01:41:13.344490+0000","last_peered":"2023-10-06T01:41:13.344490+0000","last_clean":"2023-10-06T01:41:13.344490+0000","last_became_active":"2023-10-06T01:41:13.344220+0000","last_became_peered":"2023-10-06T01:41:13.344220+0000","last_unstale":"2023-10-06T01:41:13.344490+0000","last_undegraded":"2023-10-06T01:41:13.344490+0000","last_fullsized":"2023-10-06T01:41:13.344490+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T01:51:57.229301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.345324+0000","last_change":"2023-10-06T01:41:13.345324+0000","last_active":"2023-10-06T01:41:13.345324+0000","last_peered":"2023-10-06T01:41:13.345324+0000","last_clean":"2023-10-06T01:41:13.345324+0000","last_became_active":"2023-10-06T01:41:13.344997+0000","last_became_peered":"2023-10-06T01:41:13.344997+0000","last_unstale":"2023-10-06T01:41:13.345324+0000","last_undegraded":"2023-10-06T01:41:13.345324+0000","last_fullsized":"2023-10-06T01:41:13.345324+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:27:54.074728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.355108+0000","last_change":"2023-10-06T01:41:13.355108+0000","last_active":"2023-10-06T01:41:13.355108+0000","last_peered":"2023-10-06T01:41:13.355108+0000","last_clean":"2023-10-06T01:41:13.355108+0000","last_became_active":"2023-10-06T01:41:13.354621+0000","last_became_peered":"2023-10-06T01:41:13.354621+0000","last_unstale":"2023-10-06T01:41:13.355108+0000","last_undegraded":"2023-10-06T01:41:13.355108+0000","last_fullsized":"2023-10-06T01:41:13.355108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:34:55.375989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.348956+0000","last_change":"2023-10-06T01:41:13.344094+0000","last_active":"2023-10-06T01:41:13.348956+0000","last_peered":"2023-10-06T01:41:13.348956+0000","last_clean":"2023-10-06T01:41:13.348956+0000","last_became_active":"2023-10-06T01:41:13.343796+0000","last_became_peered":"2023-10-06T01:41:13.343796+0000","last_unstale":"2023-10-06T01:41:13.348956+0000","last_undegraded":"2023-10-06T01:41:13.348956+0000","last_fullsized":"2023-10-06T01:41:13.348956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:21:30.451862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.355700+0000","last_change":"2023-10-06T01:41:13.350352+0000","last_active":"2023-10-06T01:41:13.355700+0000","last_peered":"2023-10-06T01:41:13.355700+0000","last_clean":"2023-10-06T01:41:13.355700+0000","last_became_active":"2023-10-06T01:41:13.350133+0000","last_became_peered":"2023-10-06T01:41:13.350133+0000","last_unstale":"2023-10-06T01:41:13.355700+0000","last_undegraded":"2023-10-06T01:41:13.355700+0000","last_fullsized":"2023-10-06T01:41:13.355700+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:05:37.961571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:11.306723+0000","last_change":"2023-10-06T01:41:11.306723+0000","last_active":"2023-10-06T01:41:11.306723+0000","last_peered":"2023-10-06T01:41:11.306723+0000","last_clean":"2023-10-06T01:41:11.306723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:11.306723+0000","last_undegraded":"2023-10-06T01:41:11.306723+0000","last_fullsized":"2023-10-06T01:41:11.306723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352444+0000","last_change":"2023-10-06T01:41:13.352444+0000","last_active":"2023-10-06T01:41:13.352444+0000","last_peered":"2023-10-06T01:41:13.352444+0000","last_clean":"2023-10-06T01:41:13.352444+0000","last_became_active":"2023-10-06T01:41:13.352134+0000","last_became_peered":"2023-10-06T01:41:13.352134+0000","last_unstale":"2023-10-06T01:41:13.352444+0000","last_undegraded":"2023-10-06T01:41:13.352444+0000","last_fullsized":"2023-10-06T01:41:13.352444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:11:26.794025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.336930+0000","last_change":"2023-10-06T01:41:12.347654+0000","last_active":"2023-10-06T01:41:13.336930+0000","last_peered":"2023-10-06T01:41:13.336930+0000","last_clean":"2023-10-06T01:41:13.336930+0000","last_became_active":"2023-10-06T01:41:12.347386+0000","last_became_peered":"2023-10-06T01:41:12.347386+0000","last_unstale":"2023-10-06T01:41:13.336930+0000","last_undegraded":"2023-10-06T01:41:13.336930+0000","last_fullsized":"2023-10-06T01:41:13.336930+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:07:30.529679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352350+0000","last_change":"2023-10-06T01:41:13.352350+0000","last_active":"2023-10-06T01:41:13.352350+0000","last_peered":"2023-10-06T01:41:13.352350+0000","last_clean":"2023-10-06T01:41:13.352350+0000","last_became_active":"2023-10-06T01:41:13.352091+0000","last_became_peered":"2023-10-06T01:41:13.352091+0000","last_unstale":"2023-10-06T01:41:13.352350+0000","last_undegraded":"2023-10-06T01:41:13.352350+0000","last_fullsized":"2023-10-06T01:41:13.352350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:31:20.840764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-10-06T01:41:12.349520+0000","last_change":"2023-10-06T01:41:12.343086+0000","last_active":"2023-10-06T01:41:12.349520+0000","last_peered":"2023-10-06T01:41:12.349520+0000","last_clean":"2023-10-06T01:41:12.349520+0000","last_became_active":"2023-10-06T01:41:12.342595+0000","last_became_peered":"2023-10-06T01:41:12.342595+0000","last_unstale":"2023-10-06T01:41:12.349520+0000","last_undegraded":"2023-10-06T01:41:12.349520+0000","last_fullsized":"2023-10-06T01:41:12.349520+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:15:40.841990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-10-06T01:41:12.348510+0000","last_change":"2023-10-06T01:41:12.341922+0000","last_active":"2023-10-06T01:41:12.348510+0000","last_peered":"2023-10-06T01:41:12.348510+0000","last_clean":"2023-10-06T01:41:12.348510+0000","last_became_active":"2023-10-06T01:41:12.341543+0000","last_became_peered":"2023-10-06T01:41:12.341543+0000","last_unstale":"2023-10-06T01:41:12.348510+0000","last_undegraded":"2023-10-06T01:41:12.348510+0000","last_fullsized":"2023-10-06T01:41:12.348510+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:13:14.004577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.345274+0000","last_change":"2023-10-06T01:41:13.345274+0000","last_active":"2023-10-06T01:41:13.345274+0000","last_peered":"2023-10-06T01:41:13.345274+0000","last_clean":"2023-10-06T01:41:13.345274+0000","last_became_active":"2023-10-06T01:41:13.344992+0000","last_became_peered":"2023-10-06T01:41:13.344992+0000","last_unstale":"2023-10-06T01:41:13.345274+0000","last_undegraded":"2023-10-06T01:41:13.345274+0000","last_fullsized":"2023-10-06T01:41:13.345274+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:39:03.091509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:11.306723+0000","last_change":"2023-10-06T01:41:11.306723+0000","last_active":"2023-10-06T01:41:11.306723+0000","last_peered":"2023-10-06T01:41:11.306723+0000","last_clean":"2023-10-06T01:41:11.306723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:11.306723+0000","last_undegraded":"2023-10-06T01:41:11.306723+0000","last_fullsized":"2023-10-06T01:41:11.306723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.357361+0000","last_change":"2023-10-06T01:41:13.357361+0000","last_active":"2023-10-06T01:41:13.357361+0000","last_peered":"2023-10-06T01:41:13.357361+0000","last_clean":"2023-10-06T01:41:13.357361+0000","last_became_active":"2023-10-06T01:41:13.356980+0000","last_became_peered":"2023-10-06T01:41:13.356980+0000","last_unstale":"2023-10-06T01:41:13.357361+0000","last_undegraded":"2023-10-06T01:41:13.357361+0000","last_fullsized":"2023-10-06T01:41:13.357361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:42:51.581939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:11.306723+0000","last_change":"2023-10-06T01:41:11.306723+0000","last_active":"2023-10-06T01:41:11.306723+0000","last_peered":"2023-10-06T01:41:11.306723+0000","last_clean":"2023-10-06T01:41:11.306723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:11.306723+0000","last_undegraded":"2023-10-06T01:41:11.306723+0000","last_fullsized":"2023-10-06T01:41:11.306723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352993+0000","last_change":"2023-10-06T01:41:13.348953+0000","last_active":"2023-10-06T01:41:13.352993+0000","last_peered":"2023-10-06T01:41:13.352993+0000","last_clean":"2023-10-06T01:41:13.352993+0000","last_became_active":"2023-10-06T01:41:13.348683+0000","last_became_peered":"2023-10-06T01:41:13.348683+0000","last_unstale":"2023-10-06T01:41:13.352993+0000","last_undegraded":"2023-10-06T01:41:13.352993+0000","last_fullsized":"2023-10-06T01:41:13.352993+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:42:46.214805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:11.306723+0000","last_change":"2023-10-06T01:41:11.306723+0000","last_active":"2023-10-06T01:41:11.306723+0000","last_peered":"2023-10-06T01:41:11.306723+0000","last_clean":"2023-10-06T01:41:11.306723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:11.306723+0000","last_undegraded":"2023-10-06T01:41:11.306723+0000","last_fullsized":"2023-10-06T01:41:11.306723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:11.306723+0000","last_change":"2023-10-06T01:41:11.306723+0000","last_active":"2023-10-06T01:41:11.306723+0000","last_peered":"2023-10-06T01:41:11.306723+0000","last_clean":"2023-10-06T01:41:11.306723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:11.306723+0000","last_undegraded":"2023-10-06T01:41:11.306723+0000","last_fullsized":"2023-10-06T01:41:11.306723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352574+0000","last_change":"2023-10-06T01:41:13.352574+0000","last_active":"2023-10-06T01:41:13.352574+0000","last_peered":"2023-10-06T01:41:13.352574+0000","last_clean":"2023-10-06T01:41:13.352574+0000","last_became_active":"2023-10-06T01:41:13.352300+0000","last_became_peered":"2023-10-06T01:41:13.352300+0000","last_unstale":"2023-10-06T01:41:13.352574+0000","last_undegraded":"2023-10-06T01:41:13.352574+0000","last_fullsized":"2023-10-06T01:41:13.352574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:27:54.074728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-10-06T01:41:12.351884+0000","last_change":"2023-10-06T01:41:12.344679+0000","last_active":"2023-10-06T01:41:12.351884+0000","last_peered":"2023-10-06T01:41:12.351884+0000","last_clean":"2023-10-06T01:41:12.351884+0000","last_became_active":"2023-10-06T01:41:12.344430+0000","last_became_peered":"2023-10-06T01:41:12.344430+0000","last_unstale":"2023-10-06T01:41:12.351884+0000","last_undegraded":"2023-10-06T01:41:12.351884+0000","last_fullsized":"2023-10-06T01:41:12.351884+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:27:01.582307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:11.306723+0000","last_change":"2023-10-06T01:41:11.306723+0000","last_active":"2023-10-06T01:41:11.306723+0000","last_peered":"2023-10-06T01:41:11.306723+0000","last_clean":"2023-10-06T01:41:11.306723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:11.306723+0000","last_undegraded":"2023-10-06T01:41:11.306723+0000","last_fullsized":"2023-10-06T01:41:11.306723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.343508+0000","last_change":"2023-10-06T01:41:13.343508+0000","last_active":"2023-10-06T01:41:13.343508+0000","last_peered":"2023-10-06T01:41:13.343508+0000","last_clean":"2023-10-06T01:41:13.343508+0000","last_became_active":"2023-10-06T01:41:13.343249+0000","last_became_peered":"2023-10-06T01:41:13.343249+0000","last_unstale":"2023-10-06T01:41:13.343508+0000","last_undegraded":"2023-10-06T01:41:13.343508+0000","last_fullsized":"2023-10-06T01:41:13.343508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:03:37.385158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:11.306723+0000","last_change":"2023-10-06T01:41:11.306723+0000","last_active":"2023-10-06T01:41:11.306723+0000","last_peered":"2023-10-06T01:41:11.306723+0000","last_clean":"2023-10-06T01:41:11.306723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:11.306723+0000","last_undegraded":"2023-10-06T01:41:11.306723+0000","last_fullsized":"2023-10-06T01:41:11.306723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354861+0000","last_change":"2023-10-06T01:41:13.354861+0000","last_active":"2023-10-06T01:41:13.354861+0000","last_peered":"2023-10-06T01:41:13.354861+0000","last_clean":"2023-10-06T01:41:13.354861+0000","last_became_active":"2023-10-06T01:41:13.354190+0000","last_became_peered":"2023-10-06T01:41:13.354190+0000","last_unstale":"2023-10-06T01:41:13.354861+0000","last_undegraded":"2023-10-06T01:41:13.354861+0000","last_fullsized":"2023-10-06T01:41:13.354861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:45:07.571069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:11.306723+0000","last_change":"2023-10-06T01:41:11.306723+0000","last_active":"2023-10-06T01:41:11.306723+0000","last_peered":"2023-10-06T01:41:11.306723+0000","last_clean":"2023-10-06T01:41:11.306723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:11.306723+0000","last_undegraded":"2023-10-06T01:41:11.306723+0000","last_fullsized":"2023-10-06T01:41:11.306723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:11.306723+0000","last_change":"2023-10-06T01:41:11.306723+0000","last_active":"2023-10-06T01:41:11.306723+0000","last_peered":"2023-10-06T01:41:11.306723+0000","last_clean":"2023-10-06T01:41:11.306723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:11.306723+0000","last_undegraded":"2023-10-06T01:41:11.306723+0000","last_fullsized":"2023-10-06T01:41:11.306723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:11.306723+0000","last_change":"2023-10-06T01:41:11.306723+0000","last_active":"2023-10-06T01:41:11.306723+0000","last_peered":"2023-10-06T01:41:11.306723+0000","last_clean":"2023-10-06T01:41:11.306723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:11.306723+0000","last_undegraded":"2023-10-06T01:41:11.306723+0000","last_fullsized":"2023-10-06T01:41:11.306723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:11.306723+0000","last_change":"2023-10-06T01:41:11.306723+0000","last_active":"2023-10-06T01:41:11.306723+0000","last_peered":"2023-10-06T01:41:11.306723+0000","last_clean":"2023-10-06T01:41:11.306723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:11.306723+0000","last_undegraded":"2023-10-06T01:41:11.306723+0000","last_fullsized":"2023-10-06T01:41:11.306723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.344966+0000","last_change":"2023-10-06T01:41:13.344966+0000","last_active":"2023-10-06T01:41:13.344966+0000","last_peered":"2023-10-06T01:41:13.344966+0000","last_clean":"2023-10-06T01:41:13.344966+0000","last_became_active":"2023-10-06T01:41:13.344700+0000","last_became_peered":"2023-10-06T01:41:13.344700+0000","last_unstale":"2023-10-06T01:41:13.344966+0000","last_undegraded":"2023-10-06T01:41:13.344966+0000","last_fullsized":"2023-10-06T01:41:13.344966+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:42:23.199959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"19'2","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.346245+0000","last_change":"2023-10-06T01:41:12.343169+0000","last_active":"2023-10-06T01:41:13.346245+0000","last_peered":"2023-10-06T01:41:13.346245+0000","last_clean":"2023-10-06T01:41:13.346245+0000","last_became_active":"2023-10-06T01:41:12.342474+0000","last_became_peered":"2023-10-06T01:41:12.342474+0000","last_unstale":"2023-10-06T01:41:13.346245+0000","last_undegraded":"2023-10-06T01:41:13.346245+0000","last_fullsized":"2023-10-06T01:41:13.346245+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:27:01.582307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:11.306723+0000","last_change":"2023-10-06T01:41:11.306723+0000","last_active":"2023-10-06T01:41:11.306723+0000","last_peered":"2023-10-06T01:41:11.306723+0000","last_clean":"2023-10-06T01:41:11.306723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:11.306723+0000","last_undegraded":"2023-10-06T01:41:11.306723+0000","last_fullsized":"2023-10-06T01:41:11.306723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:11.306723+0000","last_change":"2023-10-06T01:41:11.306723+0000","last_active":"2023-10-06T01:41:11.306723+0000","last_peered":"2023-10-06T01:41:11.306723+0000","last_clean":"2023-10-06T01:41:11.306723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:11.306723+0000","last_undegraded":"2023-10-06T01:41:11.306723+0000","last_fullsized":"2023-10-06T01:41:11.306723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-10-06T01:41:12.349596+0000","last_change":"2023-10-06T01:41:12.345661+0000","last_active":"2023-10-06T01:41:12.349596+0000","last_peered":"2023-10-06T01:41:12.349596+0000","last_clean":"2023-10-06T01:41:12.349596+0000","last_became_active":"2023-10-06T01:41:12.345363+0000","last_became_peered":"2023-10-06T01:41:12.345363+0000","last_unstale":"2023-10-06T01:41:12.349596+0000","last_undegraded":"2023-10-06T01:41:12.349596+0000","last_fullsized":"2023-10-06T01:41:12.349596+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:50:44.552768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356650+0000","last_change":"2023-10-06T01:41:13.353633+0000","last_active":"2023-10-06T01:41:13.356650+0000","last_peered":"2023-10-06T01:41:13.356650+0000","last_clean":"2023-10-06T01:41:13.356650+0000","last_became_active":"2023-10-06T01:41:13.353318+0000","last_became_peered":"2023-10-06T01:41:13.353318+0000","last_unstale":"2023-10-06T01:41:13.356650+0000","last_undegraded":"2023-10-06T01:41:13.356650+0000","last_fullsized":"2023-10-06T01:41:13.356650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:42:23.199959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"19'2","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354459+0000","last_change":"2023-10-06T01:41:12.340463+0000","last_active":"2023-10-06T01:41:13.354459+0000","last_peered":"2023-10-06T01:41:13.354459+0000","last_clean":"2023-10-06T01:41:13.354459+0000","last_became_active":"2023-10-06T01:41:12.340186+0000","last_became_peered":"2023-10-06T01:41:12.340186+0000","last_unstale":"2023-10-06T01:41:13.354459+0000","last_undegraded":"2023-10-06T01:41:13.354459+0000","last_fullsized":"2023-10-06T01:41:13.354459+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:16:14.826869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-06T01:41:07.454787+0000","last_change":"2023-10-06T01:41:07.454295+0000","last_active":"2023-10-06T01:41:07.454787+0000","last_peered":"2023-10-06T01:41:07.454787+0000","last_clean":"2023-10-06T01:41:07.454787+0000","last_became_active":"2023-10-06T01:41:07.454034+0000","last_became_peered":"2023-10-06T01:41:07.454034+0000","last_unstale":"2023-10-06T01:41:07.454787+0000","last_undegraded":"2023-10-06T01:41:07.454787+0000","last_fullsized":"2023-10-06T01:41:07.454787+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:32:58.914183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351289+0000","last_change":"2023-10-06T01:41:13.351289+0000","last_active":"2023-10-06T01:41:13.351289+0000","last_peered":"2023-10-06T01:41:13.351289+0000","last_clean":"2023-10-06T01:41:13.351289+0000","last_became_active":"2023-10-06T01:41:13.351022+0000","last_became_peered":"2023-10-06T01:41:13.351022+0000","last_unstale":"2023-10-06T01:41:13.351289+0000","last_undegraded":"2023-10-06T01:41:13.351289+0000","last_fullsized":"2023-10-06T01:41:13.351289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T01:51:57.229301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-10-06T01:41:12.327374+0000","last_change":"2023-10-06T01:41:07.261955+0000","last_active":"2023-10-06T01:41:12.327374+0000","last_peered":"2023-10-06T01:41:12.327374+0000","last_clean":"2023-10-06T01:41:12.327374+0000","last_became_active":"2023-10-06T01:41:07.261692+0000","last_became_peered":"2023-10-06T01:41:07.261692+0000","last_unstale":"2023-10-06T01:41:12.327374+0000","last_undegraded":"2023-10-06T01:41:12.327374+0000","last_fullsized":"2023-10-06T01:41:12.327374+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-10-06T01:41:12.344691+0000","last_change":"2023-10-06T01:41:12.341312+0000","last_active":"2023-10-06T01:41:12.344691+0000","last_peered":"2023-10-06T01:41:12.344691+0000","last_clean":"2023-10-06T01:41:12.344691+0000","last_became_active":"2023-10-06T01:41:12.340997+0000","last_became_peered":"2023-10-06T01:41:12.340997+0000","last_unstale":"2023-10-06T01:41:12.344691+0000","last_undegraded":"2023-10-06T01:41:12.344691+0000","last_fullsized":"2023-10-06T01:41:12.344691+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:06:19.029137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.336375+0000","last_change":"2023-10-06T01:41:07.258362+0000","last_active":"2023-10-06T01:41:13.336375+0000","last_peered":"2023-10-06T01:41:13.336375+0000","last_clean":"2023-10-06T01:41:13.336375+0000","last_became_active":"2023-10-06T01:41:07.258055+0000","last_became_peered":"2023-10-06T01:41:07.258055+0000","last_unstale":"2023-10-06T01:41:13.336375+0000","last_undegraded":"2023-10-06T01:41:13.336375+0000","last_fullsized":"2023-10-06T01:41:13.336375+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"16'117","reported_seq":159,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-06T01:41:08.389037+0000","last_change":"2023-10-06T01:41:07.261164+0000","last_active":"2023-10-06T01:41:08.389037+0000","last_peered":"2023-10-06T01:41:08.389037+0000","last_clean":"2023-10-06T01:41:08.389037+0000","last_became_active":"2023-10-06T01:41:07.260896+0000","last_became_peered":"2023-10-06T01:41:07.260896+0000","last_unstale":"2023-10-06T01:41:08.389037+0000","last_undegraded":"2023-10-06T01:41:08.389037+0000","last_fullsized":"2023-10-06T01:41:08.389037+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:05:22.934401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-10-06T01:41:12.348536+0000","last_change":"2023-10-06T01:41:12.343057+0000","last_active":"2023-10-06T01:41:12.348536+0000","last_peered":"2023-10-06T01:41:12.348536+0000","last_clean":"2023-10-06T01:41:12.348536+0000","last_became_active":"2023-10-06T01:41:12.342793+0000","last_became_peered":"2023-10-06T01:41:12.342793+0000","last_unstale":"2023-10-06T01:41:12.348536+0000","last_undegraded":"2023-10-06T01:41:12.348536+0000","last_fullsized":"2023-10-06T01:41:12.348536+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:15:40.841990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.346571+0000","last_change":"2023-10-06T01:41:13.346571+0000","last_active":"2023-10-06T01:41:13.346571+0000","last_peered":"2023-10-06T01:41:13.346571+0000","last_clean":"2023-10-06T01:41:13.346571+0000","last_became_active":"2023-10-06T01:41:13.346077+0000","last_became_peered":"2023-10-06T01:41:13.346077+0000","last_unstale":"2023-10-06T01:41:13.346571+0000","last_undegraded":"2023-10-06T01:41:13.346571+0000","last_fullsized":"2023-10-06T01:41:13.346571+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:42:51.581939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-06T01:41:07.461513+0000","last_change":"2023-10-06T01:41:07.456511+0000","last_active":"2023-10-06T01:41:07.461513+0000","last_peered":"2023-10-06T01:41:07.461513+0000","last_clean":"2023-10-06T01:41:07.461513+0000","last_became_active":"2023-10-06T01:41:07.456024+0000","last_became_peered":"2023-10-06T01:41:07.456024+0000","last_unstale":"2023-10-06T01:41:07.461513+0000","last_undegraded":"2023-10-06T01:41:07.461513+0000","last_fullsized":"2023-10-06T01:41:07.461513+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:51:02.949231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.336635+0000","last_change":"2023-10-06T01:41:12.343382+0000","last_active":"2023-10-06T01:41:13.336635+0000","last_peered":"2023-10-06T01:41:13.336635+0000","last_clean":"2023-10-06T01:41:13.336635+0000","last_became_active":"2023-10-06T01:41:12.343088+0000","last_became_peered":"2023-10-06T01:41:12.343088+0000","last_unstale":"2023-10-06T01:41:13.336635+0000","last_undegraded":"2023-10-06T01:41:13.336635+0000","last_fullsized":"2023-10-06T01:41:13.336635+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:44:53.014481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.350157+0000","last_change":"2023-10-06T01:41:13.346852+0000","last_active":"2023-10-06T01:41:13.350157+0000","last_peered":"2023-10-06T01:41:13.350157+0000","last_clean":"2023-10-06T01:41:13.350157+0000","last_became_active":"2023-10-06T01:41:13.346571+0000","last_became_peered":"2023-10-06T01:41:13.346571+0000","last_unstale":"2023-10-06T01:41:13.350157+0000","last_undegraded":"2023-10-06T01:41:13.350157+0000","last_fullsized":"2023-10-06T01:41:13.350157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:25:04.563235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-06T01:41:07.461501+0000","last_change":"2023-10-06T01:41:07.456273+0000","last_active":"2023-10-06T01:41:07.461501+0000","last_peered":"2023-10-06T01:41:07.461501+0000","last_clean":"2023-10-06T01:41:07.461501+0000","last_became_active":"2023-10-06T01:41:07.456000+0000","last_became_peered":"2023-10-06T01:41:07.456000+0000","last_unstale":"2023-10-06T01:41:07.461501+0000","last_undegraded":"2023-10-06T01:41:07.461501+0000","last_fullsized":"2023-10-06T01:41:07.461501+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:01:42.633388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-10-06T01:41:12.349428+0000","last_change":"2023-10-06T01:41:12.342215+0000","last_active":"2023-10-06T01:41:12.349428+0000","last_peered":"2023-10-06T01:41:12.349428+0000","last_clean":"2023-10-06T01:41:12.349428+0000","last_became_active":"2023-10-06T01:41:12.341493+0000","last_became_peered":"2023-10-06T01:41:12.341493+0000","last_unstale":"2023-10-06T01:41:12.349428+0000","last_undegraded":"2023-10-06T01:41:12.349428+0000","last_fullsized":"2023-10-06T01:41:12.349428+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:49:54.658088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-06T01:41:07.461498+0000","last_change":"2023-10-06T01:41:07.456688+0000","last_active":"2023-10-06T01:41:07.461498+0000","last_peered":"2023-10-06T01:41:07.461498+0000","last_clean":"2023-10-06T01:41:07.461498+0000","last_became_active":"2023-10-06T01:41:07.456304+0000","last_became_peered":"2023-10-06T01:41:07.456304+0000","last_unstale":"2023-10-06T01:41:07.461498+0000","last_undegraded":"2023-10-06T01:41:07.461498+0000","last_fullsized":"2023-10-06T01:41:07.461498+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:11.306723+0000","last_change":"2023-10-06T01:41:11.306723+0000","last_active":"2023-10-06T01:41:11.306723+0000","last_peered":"2023-10-06T01:41:11.306723+0000","last_clean":"2023-10-06T01:41:11.306723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:11.306723+0000","last_undegraded":"2023-10-06T01:41:11.306723+0000","last_fullsized":"2023-10-06T01:41:11.306723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352905+0000","last_change":"2023-10-06T01:41:13.348241+0000","last_active":"2023-10-06T01:41:13.352905+0000","last_peered":"2023-10-06T01:41:13.352905+0000","last_clean":"2023-10-06T01:41:13.352905+0000","last_became_active":"2023-10-06T01:41:13.347979+0000","last_became_peered":"2023-10-06T01:41:13.347979+0000","last_unstale":"2023-10-06T01:41:13.352905+0000","last_undegraded":"2023-10-06T01:41:13.352905+0000","last_fullsized":"2023-10-06T01:41:13.352905+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:41:59.186469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-06T01:41:07.459128+0000","last_change":"2023-10-06T01:41:07.456388+0000","last_active":"2023-10-06T01:41:07.459128+0000","last_peered":"2023-10-06T01:41:07.459128+0000","last_clean":"2023-10-06T01:41:07.459128+0000","last_became_active":"2023-10-06T01:41:07.456045+0000","last_became_peered":"2023-10-06T01:41:07.456045+0000","last_unstale":"2023-10-06T01:41:07.459128+0000","last_undegraded":"2023-10-06T01:41:07.459128+0000","last_fullsized":"2023-10-06T01:41:07.459128+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.338387+0000","last_change":"2023-10-06T01:41:12.344268+0000","last_active":"2023-10-06T01:41:13.338387+0000","last_peered":"2023-10-06T01:41:13.338387+0000","last_clean":"2023-10-06T01:41:13.338387+0000","last_became_active":"2023-10-06T01:41:12.343996+0000","last_became_peered":"2023-10-06T01:41:12.343996+0000","last_unstale":"2023-10-06T01:41:13.338387+0000","last_undegraded":"2023-10-06T01:41:13.338387+0000","last_fullsized":"2023-10-06T01:41:13.338387+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:37:52.143727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.346184+0000","last_change":"2023-10-06T01:41:13.346184+0000","last_active":"2023-10-06T01:41:13.346184+0000","last_peered":"2023-10-06T01:41:13.346184+0000","last_clean":"2023-10-06T01:41:13.346184+0000","last_became_active":"2023-10-06T01:41:13.345570+0000","last_became_peered":"2023-10-06T01:41:13.345570+0000","last_unstale":"2023-10-06T01:41:13.346184+0000","last_undegraded":"2023-10-06T01:41:13.346184+0000","last_fullsized":"2023-10-06T01:41:13.346184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:42:46.214805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-06T01:41:07.259421+0000","last_change":"2023-10-06T01:41:07.258162+0000","last_active":"2023-10-06T01:41:07.259421+0000","last_peered":"2023-10-06T01:41:07.259421+0000","last_clean":"2023-10-06T01:41:07.259421+0000","last_became_active":"2023-10-06T01:41:07.257795+0000","last_became_peered":"2023-10-06T01:41:07.257795+0000","last_unstale":"2023-10-06T01:41:07.259421+0000","last_undegraded":"2023-10-06T01:41:07.259421+0000","last_fullsized":"2023-10-06T01:41:07.259421+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:05:22.934401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:11.306723+0000","last_change":"2023-10-06T01:41:11.306723+0000","last_active":"2023-10-06T01:41:11.306723+0000","last_peered":"2023-10-06T01:41:11.306723+0000","last_clean":"2023-10-06T01:41:11.306723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:11.306723+0000","last_undegraded":"2023-10-06T01:41:11.306723+0000","last_fullsized":"2023-10-06T01:41:11.306723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.349056+0000","last_change":"2023-10-06T01:41:13.344589+0000","last_active":"2023-10-06T01:41:13.349056+0000","last_peered":"2023-10-06T01:41:13.349056+0000","last_clean":"2023-10-06T01:41:13.349056+0000","last_became_active":"2023-10-06T01:41:13.344281+0000","last_became_peered":"2023-10-06T01:41:13.344281+0000","last_unstale":"2023-10-06T01:41:13.349056+0000","last_undegraded":"2023-10-06T01:41:13.349056+0000","last_fullsized":"2023-10-06T01:41:13.349056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:32:17.938309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"19'1","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.329842+0000","last_change":"2023-10-06T01:41:12.347655+0000","last_active":"2023-10-06T01:41:13.329842+0000","last_peered":"2023-10-06T01:41:13.329842+0000","last_clean":"2023-10-06T01:41:13.329842+0000","last_became_active":"2023-10-06T01:41:12.347353+0000","last_became_peered":"2023-10-06T01:41:12.347353+0000","last_unstale":"2023-10-06T01:41:13.329842+0000","last_undegraded":"2023-10-06T01:41:13.329842+0000","last_fullsized":"2023-10-06T01:41:13.329842+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:49:54.658088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:11.306723+0000","last_change":"2023-10-06T01:41:11.306723+0000","last_active":"2023-10-06T01:41:11.306723+0000","last_peered":"2023-10-06T01:41:11.306723+0000","last_clean":"2023-10-06T01:41:11.306723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:11.306723+0000","last_undegraded":"2023-10-06T01:41:11.306723+0000","last_fullsized":"2023-10-06T01:41:11.306723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:11.306723+0000","last_change":"2023-10-06T01:41:11.306723+0000","last_active":"2023-10-06T01:41:11.306723+0000","last_peered":"2023-10-06T01:41:11.306723+0000","last_clean":"2023-10-06T01:41:11.306723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:11.306723+0000","last_undegraded":"2023-10-06T01:41:11.306723+0000","last_fullsized":"2023-10-06T01:41:11.306723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1e","version":"19'1","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.337057+0000","last_change":"2023-10-06T01:41:12.340776+0000","last_active":"2023-10-06T01:41:13.337057+0000","last_peered":"2023-10-06T01:41:13.337057+0000","last_clean":"2023-10-06T01:41:13.337057+0000","last_became_active":"2023-10-06T01:41:12.340294+0000","last_became_peered":"2023-10-06T01:41:12.340294+0000","last_unstale":"2023-10-06T01:41:13.337057+0000","last_undegraded":"2023-10-06T01:41:13.337057+0000","last_fullsized":"2023-10-06T01:41:13.337057+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:37:13.672716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.355273+0000","last_change":"2023-10-06T01:41:13.355273+0000","last_active":"2023-10-06T01:41:13.355273+0000","last_peered":"2023-10-06T01:41:13.355273+0000","last_clean":"2023-10-06T01:41:13.355273+0000","last_became_active":"2023-10-06T01:41:13.354977+0000","last_became_peered":"2023-10-06T01:41:13.354977+0000","last_unstale":"2023-10-06T01:41:13.355273+0000","last_undegraded":"2023-10-06T01:41:13.355273+0000","last_fullsized":"2023-10-06T01:41:13.355273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:21:42.851681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"19'4","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.345488+0000","last_change":"2023-10-06T01:41:12.338431+0000","last_active":"2023-10-06T01:41:13.345488+0000","last_peered":"2023-10-06T01:41:13.345488+0000","last_clean":"2023-10-06T01:41:13.345488+0000","last_became_active":"2023-10-06T01:41:12.338081+0000","last_became_peered":"2023-10-06T01:41:12.338081+0000","last_unstale":"2023-10-06T01:41:13.345488+0000","last_undegraded":"2023-10-06T01:41:13.345488+0000","last_fullsized":"2023-10-06T01:41:13.345488+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:16:14.826869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.355216+0000","last_change":"2023-10-06T01:41:13.355216+0000","last_active":"2023-10-06T01:41:13.355216+0000","last_peered":"2023-10-06T01:41:13.355216+0000","last_clean":"2023-10-06T01:41:13.355216+0000","last_became_active":"2023-10-06T01:41:13.354853+0000","last_became_peered":"2023-10-06T01:41:13.354853+0000","last_unstale":"2023-10-06T01:41:13.355216+0000","last_undegraded":"2023-10-06T01:41:13.355216+0000","last_fullsized":"2023-10-06T01:41:13.355216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:42:44.799267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.353822+0000","last_change":"2023-10-06T01:41:13.353822+0000","last_active":"2023-10-06T01:41:13.353822+0000","last_peered":"2023-10-06T01:41:13.353822+0000","last_clean":"2023-10-06T01:41:13.353822+0000","last_became_active":"2023-10-06T01:41:13.353292+0000","last_became_peered":"2023-10-06T01:41:13.353292+0000","last_unstale":"2023-10-06T01:41:13.353822+0000","last_undegraded":"2023-10-06T01:41:13.353822+0000","last_fullsized":"2023-10-06T01:41:13.353822+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:52:48.286318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356636+0000","last_change":"2023-10-06T01:41:13.353662+0000","last_active":"2023-10-06T01:41:13.356636+0000","last_peered":"2023-10-06T01:41:13.356636+0000","last_clean":"2023-10-06T01:41:13.356636+0000","last_became_active":"2023-10-06T01:41:13.353269+0000","last_became_peered":"2023-10-06T01:41:13.353269+0000","last_unstale":"2023-10-06T01:41:13.356636+0000","last_undegraded":"2023-10-06T01:41:13.356636+0000","last_fullsized":"2023-10-06T01:41:13.356636+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:11:44.715827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":54,"acting":54,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":554,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":30,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298584,"kb_used_data":732,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073256,"statfs":{"total":96636764160,"available":96331014144,"internally_reserved":0,"allocated":749568,"data_stored":638029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.022}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97599999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297436,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074404,"statfs":{"total":96636764160,"available":96332189696,"internally_reserved":0,"allocated":151552,"data_stored":46308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298068,"kb_used_data":780,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073772,"statfs":{"total":96636764160,"available":96331542528,"internally_reserved":0,"allocated":798720,"data_stored":664552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297476,"kb_used_data":188,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074364,"statfs":{"total":96636764160,"available":96332148736,"internally_reserved":0,"allocated":192512,"data_stored":66744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72899999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297440,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074400,"statfs":{"total":96636764160,"available":96332185600,"internally_reserved":0,"allocated":155648,"data_stored":47661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28399999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297476,"kb_used_data":188,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074364,"statfs":{"total":96636764160,"available":96332148736,"internally_reserved":0,"allocated":192512,"data_stored":66280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297440,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074400,"statfs":{"total":96636764160,"available":96332185600,"internally_reserved":0,"allocated":155648,"data_stored":47661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297436,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074404,"statfs":{"total":96636764160,"available":96332189696,"internally_reserved":0,"allocated":151552,"data_stored":46308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-06T01:41:17.453 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-06T01:41:17.606 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.606+0000 7f2f78b4b700 1 Processor -- start 2023-10-06T01:41:17.606 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.606+0000 7f2f78b4b700 1 -- start start 2023-10-06T01:41:17.607 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.607+0000 7f2f78b4b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2f7415d930 0x7f2f740d0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:17.608 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.607+0000 7f2f78b4b700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f740b2ad0 con 0x7f2f741591b0 2023-10-06T01:41:17.608 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.607+0000 7f2f78b4b700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f740b4160 con 0x7f2f7415d560 2023-10-06T01:41:17.608 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.607+0000 7f2f78b4b700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f740b2230 con 0x7f2f7415d930 2023-10-06T01:41:17.608 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.607+0000 7f2f71d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2f7415d930 0x7f2f740d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:17.608 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.607+0000 7f2f72d9d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2f741591b0 0x7f2f74159600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46202/0 (socket says 172.21.15.27:46202) 2023-10-06T01:41:17.608 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.607+0000 7f2f72d9d700 1 -- 172.21.15.27:0/1523824677 learned_addr learned my addr 172.21.15.27:0/1523824677 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:17.608 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.607+0000 7f2f71d9b700 1 -- 172.21.15.27:0/1523824677 >> v1:172.21.15.124:6789/0 conn(0x7f2f7415d560 legacy=0x7f2f740b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:17.608 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.607+0000 7f2f71d9b700 1 -- 172.21.15.27:0/1523824677 >> v1:172.21.15.27:6789/0 conn(0x7f2f741591b0 legacy=0x7f2f74159600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:17.608 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.607+0000 7f2f71d9b700 1 -- 172.21.15.27:0/1523824677 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f740b2510 con 0x7f2f7415d930 2023-10-06T01:41:17.608 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.607+0000 7f2f7159a700 1 -- 172.21.15.27:0/1523824677 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 329011124 0 0) 0x7f2f740b2ad0 con 0x7f2f741591b0 2023-10-06T01:41:17.608 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.607+0000 7f2f71d9b700 1 --2- 172.21.15.27:0/1523824677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f7415d930 0x7f2f740d0030 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c0098e0 tx=0x7f2f5c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=206a45c4c06c5cd9 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:17.608 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.607+0000 7f2f7159a700 1 -- 172.21.15.27:0/1523824677 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5c009e80 con 0x7f2f7415d930 2023-10-06T01:41:17.609 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.607+0000 7f2f7159a700 1 -- 172.21.15.27:0/1523824677 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f5c00b860 con 0x7f2f7415d930 2023-10-06T01:41:17.609 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.607+0000 7f2f7159a700 1 -- 172.21.15.27:0/1523824677 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5c00bbe0 con 0x7f2f7415d930 2023-10-06T01:41:17.609 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.608+0000 7f2f78b4b700 1 -- 172.21.15.27:0/1523824677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f7415d930 msgr2=0x7f2f740d0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:17.610 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.608+0000 7f2f78b4b700 1 --2- 172.21.15.27:0/1523824677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f7415d930 0x7f2f740d0030 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c0098e0 tx=0x7f2f5c0091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:17.610 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.608+0000 7f2f78b4b700 1 -- 172.21.15.27:0/1523824677 shutdown_connections 2023-10-06T01:41:17.610 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.608+0000 7f2f78b4b700 1 --2- 172.21.15.27:0/1523824677 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f7415d930 0x7f2f740d0030 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:17.610 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.608+0000 7f2f78b4b700 1 -- 172.21.15.27:0/1523824677 >> 172.21.15.27:0/1523824677 conn(0x7f2f740d1180 msgr2=0x7f2f74069640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:17.610 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.608+0000 7f2f78b4b700 1 -- 172.21.15.27:0/1523824677 shutdown_connections 2023-10-06T01:41:17.610 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.608+0000 7f2f78b4b700 1 -- 172.21.15.27:0/1523824677 wait complete. 2023-10-06T01:41:17.610 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.609+0000 7f2f78b4b700 1 Processor -- start 2023-10-06T01:41:17.610 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.609+0000 7f2f78b4b700 1 -- start start 2023-10-06T01:41:17.612 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.610+0000 7f2f78b4b700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2f741591b0 0x7f2f74156120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:17.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.610+0000 7f2f78b4b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f7415d560 0x7f2f7415c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:17.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.610+0000 7f2f78b4b700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2f7415d930 0x7f2f7415cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:17.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.610+0000 7f2f78b4b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f74156660 con 0x7f2f7415d560 2023-10-06T01:41:17.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.610+0000 7f2f78b4b700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f740a9840 con 0x7f2f7415d930 2023-10-06T01:41:17.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.610+0000 7f2f78b4b700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f740a9b20 con 0x7f2f741591b0 2023-10-06T01:41:17.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.610+0000 7f2f71d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f7415d560 0x7f2f7415c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:17.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.610+0000 7f2f71d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f7415d560 0x7f2f7415c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55776/0 (socket says 172.21.15.27:55776) 2023-10-06T01:41:17.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.610+0000 7f2f71d9b700 1 -- 172.21.15.27:0/3165976137 learned_addr learned my addr 172.21.15.27:0/3165976137 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:17.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.610+0000 7f2f72d9d700 1 --2- 172.21.15.27:0/3165976137 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2f7415d930 0x7f2f7415cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:17.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.610+0000 7f2f71d9b700 1 -- 172.21.15.27:0/3165976137 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2f741591b0 msgr2=0x7f2f74156120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:17.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.610+0000 7f2f7259c700 1 --2- 172.21.15.27:0/3165976137 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2f741591b0 0x7f2f74156120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:17.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.610+0000 7f2f71d9b700 1 --2- 172.21.15.27:0/3165976137 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2f741591b0 0x7f2f74156120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:17.614 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.610+0000 7f2f71d9b700 1 -- 172.21.15.27:0/3165976137 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2f7415d930 msgr2=0x7f2f7415cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:17.614 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.610+0000 7f2f71d9b700 1 --2- 172.21.15.27:0/3165976137 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2f7415d930 0x7f2f7415cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:17.614 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.610+0000 7f2f71d9b700 1 -- 172.21.15.27:0/3165976137 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f5c009420 con 0x7f2f7415d560 2023-10-06T01:41:17.614 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.610+0000 7f2f71d9b700 1 --2- 172.21.15.27:0/3165976137 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f7415d560 0x7f2f7415c5e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2f74064a20 tx=0x7f2f5c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:17.614 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.611+0000 7f2f6a7fc700 1 -- 172.21.15.27:0/3165976137 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5c01f070 con 0x7f2f7415d560 2023-10-06T01:41:17.614 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.611+0000 7f2f6a7fc700 1 -- 172.21.15.27:0/3165976137 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f5c000cc0 con 0x7f2f7415d560 2023-10-06T01:41:17.614 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.611+0000 7f2f78b4b700 1 -- 172.21.15.27:0/3165976137 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f740a9e00 con 0x7f2f7415d560 2023-10-06T01:41:17.614 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.611+0000 7f2f6a7fc700 1 -- 172.21.15.27:0/3165976137 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5c00bbe0 con 0x7f2f7415d560 2023-10-06T01:41:17.614 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.611+0000 7f2f78b4b700 1 -- 172.21.15.27:0/3165976137 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f740ad080 con 0x7f2f7415d560 2023-10-06T01:41:17.614 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.611+0000 7f2f78b4b700 1 -- 172.21.15.27:0/3165976137 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f740ad6a0 con 0x7f2f7415d560 2023-10-06T01:41:17.614 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.612+0000 7f2f6a7fc700 1 -- 172.21.15.27:0/3165976137 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2f5c0118c0 con 0x7f2f7415d560 2023-10-06T01:41:17.615 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.612+0000 7f2f6a7fc700 1 --2- 172.21.15.27:0/3165976137 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2f54066f30 0x7f2f540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:17.615 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.612+0000 7f2f6a7fc700 1 -- 172.21.15.27:0/3165976137 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f2f5c085bb0 con 0x7f2f7415d560 2023-10-06T01:41:17.615 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.613+0000 7f2f7259c700 1 --2- 172.21.15.27:0/3165976137 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2f54066f30 0x7f2f540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:17.615 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.613+0000 7f2f7259c700 1 --2- 172.21.15.27:0/3165976137 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2f54066f30 0x7f2f540693e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c007fd0 tx=0x7f2f6c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:17.616 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.617+0000 7f2f6a7fc700 1 -- 172.21.15.27:0/3165976137 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2f740ad6a0 con 0x7f2f7415d560 2023-10-06T01:41:17.804 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.803+0000 7f2f78b4b700 1 -- 172.21.15.27:0/3165976137 --> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2f740ad6a0 con 0x7f2f54066f30 2023-10-06T01:41:17.811 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.811+0000 7f2f6a7fc700 1 -- 172.21.15.27:0/3165976137 <== mgr.4100 v2:172.21.15.124:6832/90278 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+278408 (secure 0 0 0) 0x7f2f740ad6a0 con 0x7f2f54066f30 2023-10-06T01:41:17.815 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:17.817 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.814+0000 7f2f78b4b700 1 -- 172.21.15.27:0/3165976137 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2f54066f30 msgr2=0x7f2f540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:17.817 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.814+0000 7f2f78b4b700 1 --2- 172.21.15.27:0/3165976137 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2f54066f30 0x7f2f540693e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2f6c007fd0 tx=0x7f2f6c009040 comp rx=0 tx=0).stop 2023-10-06T01:41:17.817 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.814+0000 7f2f78b4b700 1 -- 172.21.15.27:0/3165976137 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f7415d560 msgr2=0x7f2f7415c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:17.817 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.814+0000 7f2f78b4b700 1 --2- 172.21.15.27:0/3165976137 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f7415d560 0x7f2f7415c5e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2f74064a20 tx=0x7f2f5c017000 comp rx=0 tx=0).stop 2023-10-06T01:41:17.817 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.814+0000 7f2f78b4b700 1 -- 172.21.15.27:0/3165976137 shutdown_connections 2023-10-06T01:41:17.817 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.814+0000 7f2f78b4b700 1 --2- 172.21.15.27:0/3165976137 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2f54066f30 0x7f2f540693e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:17.817 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.814+0000 7f2f78b4b700 1 --2- 172.21.15.27:0/3165976137 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2f741591b0 0x7f2f74156120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:17.817 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.814+0000 7f2f78b4b700 1 --2- 172.21.15.27:0/3165976137 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2f7415d930 0x7f2f7415cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:17.817 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.814+0000 7f2f78b4b700 1 --2- 172.21.15.27:0/3165976137 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f7415d560 0x7f2f7415c5e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:17.817 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.814+0000 7f2f78b4b700 1 -- 172.21.15.27:0/3165976137 >> 172.21.15.27:0/3165976137 conn(0x7f2f740d1180 msgr2=0x7f2f740682f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:17.818 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.814+0000 7f2f78b4b700 1 -- 172.21.15.27:0/3165976137 shutdown_connections 2023-10-06T01:41:17.818 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.814+0000 7f2f78b4b700 1 -- 172.21.15.27:0/3165976137 wait complete. 2023-10-06T01:41:17.818 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-06T01:41:17.826 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2023-10-06T01:41:16.038592+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590922,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":167,"num_write_kb":2417,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":127,"ondisk_log_size":127,"up":102,"acting":102,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":52,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2381356,"kb_used_data":2488,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752593364,"statfs":{"total":773094113280,"available":770655604736,"internally_reserved":0,"allocated":2547712,"data_stored":1623543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":554,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.722059"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.342002+0000","last_change":"2023-10-06T01:41:13.342002+0000","last_active":"2023-10-06T01:41:13.342002+0000","last_peered":"2023-10-06T01:41:13.342002+0000","last_clean":"2023-10-06T01:41:13.342002+0000","last_became_active":"2023-10-06T01:41:13.341403+0000","last_became_peered":"2023-10-06T01:41:13.341403+0000","last_unstale":"2023-10-06T01:41:13.342002+0000","last_undegraded":"2023-10-06T01:41:13.342002+0000","last_fullsized":"2023-10-06T01:41:13.342002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:34:55.375989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.343792+0000","last_change":"2023-10-06T01:41:13.343792+0000","last_active":"2023-10-06T01:41:13.343792+0000","last_peered":"2023-10-06T01:41:13.343792+0000","last_clean":"2023-10-06T01:41:13.343792+0000","last_became_active":"2023-10-06T01:41:13.343432+0000","last_became_peered":"2023-10-06T01:41:13.343432+0000","last_unstale":"2023-10-06T01:41:13.343792+0000","last_undegraded":"2023-10-06T01:41:13.343792+0000","last_fullsized":"2023-10-06T01:41:13.343792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:52:48.286318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.344490+0000","last_change":"2023-10-06T01:41:13.344490+0000","last_active":"2023-10-06T01:41:13.344490+0000","last_peered":"2023-10-06T01:41:13.344490+0000","last_clean":"2023-10-06T01:41:13.344490+0000","last_became_active":"2023-10-06T01:41:13.344220+0000","last_became_peered":"2023-10-06T01:41:13.344220+0000","last_unstale":"2023-10-06T01:41:13.344490+0000","last_undegraded":"2023-10-06T01:41:13.344490+0000","last_fullsized":"2023-10-06T01:41:13.344490+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T01:51:57.229301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.345324+0000","last_change":"2023-10-06T01:41:13.345324+0000","last_active":"2023-10-06T01:41:13.345324+0000","last_peered":"2023-10-06T01:41:13.345324+0000","last_clean":"2023-10-06T01:41:13.345324+0000","last_became_active":"2023-10-06T01:41:13.344997+0000","last_became_peered":"2023-10-06T01:41:13.344997+0000","last_unstale":"2023-10-06T01:41:13.345324+0000","last_undegraded":"2023-10-06T01:41:13.345324+0000","last_fullsized":"2023-10-06T01:41:13.345324+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:27:54.074728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.355108+0000","last_change":"2023-10-06T01:41:13.355108+0000","last_active":"2023-10-06T01:41:13.355108+0000","last_peered":"2023-10-06T01:41:13.355108+0000","last_clean":"2023-10-06T01:41:13.355108+0000","last_became_active":"2023-10-06T01:41:13.354621+0000","last_became_peered":"2023-10-06T01:41:13.354621+0000","last_unstale":"2023-10-06T01:41:13.355108+0000","last_undegraded":"2023-10-06T01:41:13.355108+0000","last_fullsized":"2023-10-06T01:41:13.355108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:34:55.375989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.348956+0000","last_change":"2023-10-06T01:41:13.344094+0000","last_active":"2023-10-06T01:41:13.348956+0000","last_peered":"2023-10-06T01:41:13.348956+0000","last_clean":"2023-10-06T01:41:13.348956+0000","last_became_active":"2023-10-06T01:41:13.343796+0000","last_became_peered":"2023-10-06T01:41:13.343796+0000","last_unstale":"2023-10-06T01:41:13.348956+0000","last_undegraded":"2023-10-06T01:41:13.348956+0000","last_fullsized":"2023-10-06T01:41:13.348956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:21:30.451862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.355700+0000","last_change":"2023-10-06T01:41:13.350352+0000","last_active":"2023-10-06T01:41:13.355700+0000","last_peered":"2023-10-06T01:41:13.355700+0000","last_clean":"2023-10-06T01:41:13.355700+0000","last_became_active":"2023-10-06T01:41:13.350133+0000","last_became_peered":"2023-10-06T01:41:13.350133+0000","last_unstale":"2023-10-06T01:41:13.355700+0000","last_undegraded":"2023-10-06T01:41:13.355700+0000","last_fullsized":"2023-10-06T01:41:13.355700+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:05:37.961571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:11.306723+0000","last_change":"2023-10-06T01:41:11.306723+0000","last_active":"2023-10-06T01:41:11.306723+0000","last_peered":"2023-10-06T01:41:11.306723+0000","last_clean":"2023-10-06T01:41:11.306723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:11.306723+0000","last_undegraded":"2023-10-06T01:41:11.306723+0000","last_fullsized":"2023-10-06T01:41:11.306723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352444+0000","last_change":"2023-10-06T01:41:13.352444+0000","last_active":"2023-10-06T01:41:13.352444+0000","last_peered":"2023-10-06T01:41:13.352444+0000","last_clean":"2023-10-06T01:41:13.352444+0000","last_became_active":"2023-10-06T01:41:13.352134+0000","last_became_peered":"2023-10-06T01:41:13.352134+0000","last_unstale":"2023-10-06T01:41:13.352444+0000","last_undegraded":"2023-10-06T01:41:13.352444+0000","last_fullsized":"2023-10-06T01:41:13.352444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:11:26.794025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.336930+0000","last_change":"2023-10-06T01:41:12.347654+0000","last_active":"2023-10-06T01:41:13.336930+0000","last_peered":"2023-10-06T01:41:13.336930+0000","last_clean":"2023-10-06T01:41:13.336930+0000","last_became_active":"2023-10-06T01:41:12.347386+0000","last_became_peered":"2023-10-06T01:41:12.347386+0000","last_unstale":"2023-10-06T01:41:13.336930+0000","last_undegraded":"2023-10-06T01:41:13.336930+0000","last_fullsized":"2023-10-06T01:41:13.336930+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:07:30.529679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352350+0000","last_change":"2023-10-06T01:41:13.352350+0000","last_active":"2023-10-06T01:41:13.352350+0000","last_peered":"2023-10-06T01:41:13.352350+0000","last_clean":"2023-10-06T01:41:13.352350+0000","last_became_active":"2023-10-06T01:41:13.352091+0000","last_became_peered":"2023-10-06T01:41:13.352091+0000","last_unstale":"2023-10-06T01:41:13.352350+0000","last_undegraded":"2023-10-06T01:41:13.352350+0000","last_fullsized":"2023-10-06T01:41:13.352350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:31:20.840764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-10-06T01:41:12.349520+0000","last_change":"2023-10-06T01:41:12.343086+0000","last_active":"2023-10-06T01:41:12.349520+0000","last_peered":"2023-10-06T01:41:12.349520+0000","last_clean":"2023-10-06T01:41:12.349520+0000","last_became_active":"2023-10-06T01:41:12.342595+0000","last_became_peered":"2023-10-06T01:41:12.342595+0000","last_unstale":"2023-10-06T01:41:12.349520+0000","last_undegraded":"2023-10-06T01:41:12.349520+0000","last_fullsized":"2023-10-06T01:41:12.349520+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:15:40.841990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-10-06T01:41:12.348510+0000","last_change":"2023-10-06T01:41:12.341922+0000","last_active":"2023-10-06T01:41:12.348510+0000","last_peered":"2023-10-06T01:41:12.348510+0000","last_clean":"2023-10-06T01:41:12.348510+0000","last_became_active":"2023-10-06T01:41:12.341543+0000","last_became_peered":"2023-10-06T01:41:12.341543+0000","last_unstale":"2023-10-06T01:41:12.348510+0000","last_undegraded":"2023-10-06T01:41:12.348510+0000","last_fullsized":"2023-10-06T01:41:12.348510+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:13:14.004577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.345274+0000","last_change":"2023-10-06T01:41:13.345274+0000","last_active":"2023-10-06T01:41:13.345274+0000","last_peered":"2023-10-06T01:41:13.345274+0000","last_clean":"2023-10-06T01:41:13.345274+0000","last_became_active":"2023-10-06T01:41:13.344992+0000","last_became_peered":"2023-10-06T01:41:13.344992+0000","last_unstale":"2023-10-06T01:41:13.345274+0000","last_undegraded":"2023-10-06T01:41:13.345274+0000","last_fullsized":"2023-10-06T01:41:13.345274+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:39:03.091509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:11.306723+0000","last_change":"2023-10-06T01:41:11.306723+0000","last_active":"2023-10-06T01:41:11.306723+0000","last_peered":"2023-10-06T01:41:11.306723+0000","last_clean":"2023-10-06T01:41:11.306723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:11.306723+0000","last_undegraded":"2023-10-06T01:41:11.306723+0000","last_fullsized":"2023-10-06T01:41:11.306723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.357361+0000","last_change":"2023-10-06T01:41:13.357361+0000","last_active":"2023-10-06T01:41:13.357361+0000","last_peered":"2023-10-06T01:41:13.357361+0000","last_clean":"2023-10-06T01:41:13.357361+0000","last_became_active":"2023-10-06T01:41:13.356980+0000","last_became_peered":"2023-10-06T01:41:13.356980+0000","last_unstale":"2023-10-06T01:41:13.357361+0000","last_undegraded":"2023-10-06T01:41:13.357361+0000","last_fullsized":"2023-10-06T01:41:13.357361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:42:51.581939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:11.306723+0000","last_change":"2023-10-06T01:41:11.306723+0000","last_active":"2023-10-06T01:41:11.306723+0000","last_peered":"2023-10-06T01:41:11.306723+0000","last_clean":"2023-10-06T01:41:11.306723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:11.306723+0000","last_undegraded":"2023-10-06T01:41:11.306723+0000","last_fullsized":"2023-10-06T01:41:11.306723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352993+0000","last_change":"2023-10-06T01:41:13.348953+0000","last_active":"2023-10-06T01:41:13.352993+0000","last_peered":"2023-10-06T01:41:13.352993+0000","last_clean":"2023-10-06T01:41:13.352993+0000","last_became_active":"2023-10-06T01:41:13.348683+0000","last_became_peered":"2023-10-06T01:41:13.348683+0000","last_unstale":"2023-10-06T01:41:13.352993+0000","last_undegraded":"2023-10-06T01:41:13.352993+0000","last_fullsized":"2023-10-06T01:41:13.352993+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:42:46.214805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:11.306723+0000","last_change":"2023-10-06T01:41:11.306723+0000","last_active":"2023-10-06T01:41:11.306723+0000","last_peered":"2023-10-06T01:41:11.306723+0000","last_clean":"2023-10-06T01:41:11.306723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:11.306723+0000","last_undegraded":"2023-10-06T01:41:11.306723+0000","last_fullsized":"2023-10-06T01:41:11.306723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:11.306723+0000","last_change":"2023-10-06T01:41:11.306723+0000","last_active":"2023-10-06T01:41:11.306723+0000","last_peered":"2023-10-06T01:41:11.306723+0000","last_clean":"2023-10-06T01:41:11.306723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:11.306723+0000","last_undegraded":"2023-10-06T01:41:11.306723+0000","last_fullsized":"2023-10-06T01:41:11.306723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352574+0000","last_change":"2023-10-06T01:41:13.352574+0000","last_active":"2023-10-06T01:41:13.352574+0000","last_peered":"2023-10-06T01:41:13.352574+0000","last_clean":"2023-10-06T01:41:13.352574+0000","last_became_active":"2023-10-06T01:41:13.352300+0000","last_became_peered":"2023-10-06T01:41:13.352300+0000","last_unstale":"2023-10-06T01:41:13.352574+0000","last_undegraded":"2023-10-06T01:41:13.352574+0000","last_fullsized":"2023-10-06T01:41:13.352574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:27:54.074728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-10-06T01:41:12.351884+0000","last_change":"2023-10-06T01:41:12.344679+0000","last_active":"2023-10-06T01:41:12.351884+0000","last_peered":"2023-10-06T01:41:12.351884+0000","last_clean":"2023-10-06T01:41:12.351884+0000","last_became_active":"2023-10-06T01:41:12.344430+0000","last_became_peered":"2023-10-06T01:41:12.344430+0000","last_unstale":"2023-10-06T01:41:12.351884+0000","last_undegraded":"2023-10-06T01:41:12.351884+0000","last_fullsized":"2023-10-06T01:41:12.351884+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:27:01.582307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:11.306723+0000","last_change":"2023-10-06T01:41:11.306723+0000","last_active":"2023-10-06T01:41:11.306723+0000","last_peered":"2023-10-06T01:41:11.306723+0000","last_clean":"2023-10-06T01:41:11.306723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:11.306723+0000","last_undegraded":"2023-10-06T01:41:11.306723+0000","last_fullsized":"2023-10-06T01:41:11.306723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.343508+0000","last_change":"2023-10-06T01:41:13.343508+0000","last_active":"2023-10-06T01:41:13.343508+0000","last_peered":"2023-10-06T01:41:13.343508+0000","last_clean":"2023-10-06T01:41:13.343508+0000","last_became_active":"2023-10-06T01:41:13.343249+0000","last_became_peered":"2023-10-06T01:41:13.343249+0000","last_unstale":"2023-10-06T01:41:13.343508+0000","last_undegraded":"2023-10-06T01:41:13.343508+0000","last_fullsized":"2023-10-06T01:41:13.343508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:03:37.385158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:11.306723+0000","last_change":"2023-10-06T01:41:11.306723+0000","last_active":"2023-10-06T01:41:11.306723+0000","last_peered":"2023-10-06T01:41:11.306723+0000","last_clean":"2023-10-06T01:41:11.306723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:11.306723+0000","last_undegraded":"2023-10-06T01:41:11.306723+0000","last_fullsized":"2023-10-06T01:41:11.306723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354861+0000","last_change":"2023-10-06T01:41:13.354861+0000","last_active":"2023-10-06T01:41:13.354861+0000","last_peered":"2023-10-06T01:41:13.354861+0000","last_clean":"2023-10-06T01:41:13.354861+0000","last_became_active":"2023-10-06T01:41:13.354190+0000","last_became_peered":"2023-10-06T01:41:13.354190+0000","last_unstale":"2023-10-06T01:41:13.354861+0000","last_undegraded":"2023-10-06T01:41:13.354861+0000","last_fullsized":"2023-10-06T01:41:13.354861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:45:07.571069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:11.306723+0000","last_change":"2023-10-06T01:41:11.306723+0000","last_active":"2023-10-06T01:41:11.306723+0000","last_peered":"2023-10-06T01:41:11.306723+0000","last_clean":"2023-10-06T01:41:11.306723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:11.306723+0000","last_undegraded":"2023-10-06T01:41:11.306723+0000","last_fullsized":"2023-10-06T01:41:11.306723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:11.306723+0000","last_change":"2023-10-06T01:41:11.306723+0000","last_active":"2023-10-06T01:41:11.306723+0000","last_peered":"2023-10-06T01:41:11.306723+0000","last_clean":"2023-10-06T01:41:11.306723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:11.306723+0000","last_undegraded":"2023-10-06T01:41:11.306723+0000","last_fullsized":"2023-10-06T01:41:11.306723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:11.306723+0000","last_change":"2023-10-06T01:41:11.306723+0000","last_active":"2023-10-06T01:41:11.306723+0000","last_peered":"2023-10-06T01:41:11.306723+0000","last_clean":"2023-10-06T01:41:11.306723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:11.306723+0000","last_undegraded":"2023-10-06T01:41:11.306723+0000","last_fullsized":"2023-10-06T01:41:11.306723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:11.306723+0000","last_change":"2023-10-06T01:41:11.306723+0000","last_active":"2023-10-06T01:41:11.306723+0000","last_peered":"2023-10-06T01:41:11.306723+0000","last_clean":"2023-10-06T01:41:11.306723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:11.306723+0000","last_undegraded":"2023-10-06T01:41:11.306723+0000","last_fullsized":"2023-10-06T01:41:11.306723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.344966+0000","last_change":"2023-10-06T01:41:13.344966+0000","last_active":"2023-10-06T01:41:13.344966+0000","last_peered":"2023-10-06T01:41:13.344966+0000","last_clean":"2023-10-06T01:41:13.344966+0000","last_became_active":"2023-10-06T01:41:13.344700+0000","last_became_peered":"2023-10-06T01:41:13.344700+0000","last_unstale":"2023-10-06T01:41:13.344966+0000","last_undegraded":"2023-10-06T01:41:13.344966+0000","last_fullsized":"2023-10-06T01:41:13.344966+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:42:23.199959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"19'2","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.346245+0000","last_change":"2023-10-06T01:41:12.343169+0000","last_active":"2023-10-06T01:41:13.346245+0000","last_peered":"2023-10-06T01:41:13.346245+0000","last_clean":"2023-10-06T01:41:13.346245+0000","last_became_active":"2023-10-06T01:41:12.342474+0000","last_became_peered":"2023-10-06T01:41:12.342474+0000","last_unstale":"2023-10-06T01:41:13.346245+0000","last_undegraded":"2023-10-06T01:41:13.346245+0000","last_fullsized":"2023-10-06T01:41:13.346245+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:27:01.582307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:11.306723+0000","last_change":"2023-10-06T01:41:11.306723+0000","last_active":"2023-10-06T01:41:11.306723+0000","last_peered":"2023-10-06T01:41:11.306723+0000","last_clean":"2023-10-06T01:41:11.306723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:11.306723+0000","last_undegraded":"2023-10-06T01:41:11.306723+0000","last_fullsized":"2023-10-06T01:41:11.306723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:11.306723+0000","last_change":"2023-10-06T01:41:11.306723+0000","last_active":"2023-10-06T01:41:11.306723+0000","last_peered":"2023-10-06T01:41:11.306723+0000","last_clean":"2023-10-06T01:41:11.306723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:11.306723+0000","last_undegraded":"2023-10-06T01:41:11.306723+0000","last_fullsized":"2023-10-06T01:41:11.306723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-10-06T01:41:12.349596+0000","last_change":"2023-10-06T01:41:12.345661+0000","last_active":"2023-10-06T01:41:12.349596+0000","last_peered":"2023-10-06T01:41:12.349596+0000","last_clean":"2023-10-06T01:41:12.349596+0000","last_became_active":"2023-10-06T01:41:12.345363+0000","last_became_peered":"2023-10-06T01:41:12.345363+0000","last_unstale":"2023-10-06T01:41:12.349596+0000","last_undegraded":"2023-10-06T01:41:12.349596+0000","last_fullsized":"2023-10-06T01:41:12.349596+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:50:44.552768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356650+0000","last_change":"2023-10-06T01:41:13.353633+0000","last_active":"2023-10-06T01:41:13.356650+0000","last_peered":"2023-10-06T01:41:13.356650+0000","last_clean":"2023-10-06T01:41:13.356650+0000","last_became_active":"2023-10-06T01:41:13.353318+0000","last_became_peered":"2023-10-06T01:41:13.353318+0000","last_unstale":"2023-10-06T01:41:13.356650+0000","last_undegraded":"2023-10-06T01:41:13.356650+0000","last_fullsized":"2023-10-06T01:41:13.356650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:42:23.199959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"19'2","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354459+0000","last_change":"2023-10-06T01:41:12.340463+0000","last_active":"2023-10-06T01:41:13.354459+0000","last_peered":"2023-10-06T01:41:13.354459+0000","last_clean":"2023-10-06T01:41:13.354459+0000","last_became_active":"2023-10-06T01:41:12.340186+0000","last_became_peered":"2023-10-06T01:41:12.340186+0000","last_unstale":"2023-10-06T01:41:13.354459+0000","last_undegraded":"2023-10-06T01:41:13.354459+0000","last_fullsized":"2023-10-06T01:41:13.354459+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:16:14.826869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-06T01:41:07.454787+0000","last_change":"2023-10-06T01:41:07.454295+0000","last_active":"2023-10-06T01:41:07.454787+0000","last_peered":"2023-10-06T01:41:07.454787+0000","last_clean":"2023-10-06T01:41:07.454787+0000","last_became_active":"2023-10-06T01:41:07.454034+0000","last_became_peered":"2023-10-06T01:41:07.454034+0000","last_unstale":"2023-10-06T01:41:07.454787+0000","last_undegraded":"2023-10-06T01:41:07.454787+0000","last_fullsized":"2023-10-06T01:41:07.454787+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:32:58.914183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351289+0000","last_change":"2023-10-06T01:41:13.351289+0000","last_active":"2023-10-06T01:41:13.351289+0000","last_peered":"2023-10-06T01:41:13.351289+0000","last_clean":"2023-10-06T01:41:13.351289+0000","last_became_active":"2023-10-06T01:41:13.351022+0000","last_became_peered":"2023-10-06T01:41:13.351022+0000","last_unstale":"2023-10-06T01:41:13.351289+0000","last_undegraded":"2023-10-06T01:41:13.351289+0000","last_fullsized":"2023-10-06T01:41:13.351289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T01:51:57.229301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-10-06T01:41:12.327374+0000","last_change":"2023-10-06T01:41:07.261955+0000","last_active":"2023-10-06T01:41:12.327374+0000","last_peered":"2023-10-06T01:41:12.327374+0000","last_clean":"2023-10-06T01:41:12.327374+0000","last_became_active":"2023-10-06T01:41:07.261692+0000","last_became_peered":"2023-10-06T01:41:07.261692+0000","last_unstale":"2023-10-06T01:41:12.327374+0000","last_undegraded":"2023-10-06T01:41:12.327374+0000","last_fullsized":"2023-10-06T01:41:12.327374+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-10-06T01:41:12.344691+0000","last_change":"2023-10-06T01:41:12.341312+0000","last_active":"2023-10-06T01:41:12.344691+0000","last_peered":"2023-10-06T01:41:12.344691+0000","last_clean":"2023-10-06T01:41:12.344691+0000","last_became_active":"2023-10-06T01:41:12.340997+0000","last_became_peered":"2023-10-06T01:41:12.340997+0000","last_unstale":"2023-10-06T01:41:12.344691+0000","last_undegraded":"2023-10-06T01:41:12.344691+0000","last_fullsized":"2023-10-06T01:41:12.344691+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:06:19.029137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.336375+0000","last_change":"2023-10-06T01:41:07.258362+0000","last_active":"2023-10-06T01:41:13.336375+0000","last_peered":"2023-10-06T01:41:13.336375+0000","last_clean":"2023-10-06T01:41:13.336375+0000","last_became_active":"2023-10-06T01:41:07.258055+0000","last_became_peered":"2023-10-06T01:41:07.258055+0000","last_unstale":"2023-10-06T01:41:13.336375+0000","last_undegraded":"2023-10-06T01:41:13.336375+0000","last_fullsized":"2023-10-06T01:41:13.336375+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"16'117","reported_seq":159,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-10-06T01:41:08.389037+0000","last_change":"2023-10-06T01:41:07.261164+0000","last_active":"2023-10-06T01:41:08.389037+0000","last_peered":"2023-10-06T01:41:08.389037+0000","last_clean":"2023-10-06T01:41:08.389037+0000","last_became_active":"2023-10-06T01:41:07.260896+0000","last_became_peered":"2023-10-06T01:41:07.260896+0000","last_unstale":"2023-10-06T01:41:08.389037+0000","last_undegraded":"2023-10-06T01:41:08.389037+0000","last_fullsized":"2023-10-06T01:41:08.389037+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:05:22.934401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-10-06T01:41:12.348536+0000","last_change":"2023-10-06T01:41:12.343057+0000","last_active":"2023-10-06T01:41:12.348536+0000","last_peered":"2023-10-06T01:41:12.348536+0000","last_clean":"2023-10-06T01:41:12.348536+0000","last_became_active":"2023-10-06T01:41:12.342793+0000","last_became_peered":"2023-10-06T01:41:12.342793+0000","last_unstale":"2023-10-06T01:41:12.348536+0000","last_undegraded":"2023-10-06T01:41:12.348536+0000","last_fullsized":"2023-10-06T01:41:12.348536+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:15:40.841990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.346571+0000","last_change":"2023-10-06T01:41:13.346571+0000","last_active":"2023-10-06T01:41:13.346571+0000","last_peered":"2023-10-06T01:41:13.346571+0000","last_clean":"2023-10-06T01:41:13.346571+0000","last_became_active":"2023-10-06T01:41:13.346077+0000","last_became_peered":"2023-10-06T01:41:13.346077+0000","last_unstale":"2023-10-06T01:41:13.346571+0000","last_undegraded":"2023-10-06T01:41:13.346571+0000","last_fullsized":"2023-10-06T01:41:13.346571+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:42:51.581939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-06T01:41:07.461513+0000","last_change":"2023-10-06T01:41:07.456511+0000","last_active":"2023-10-06T01:41:07.461513+0000","last_peered":"2023-10-06T01:41:07.461513+0000","last_clean":"2023-10-06T01:41:07.461513+0000","last_became_active":"2023-10-06T01:41:07.456024+0000","last_became_peered":"2023-10-06T01:41:07.456024+0000","last_unstale":"2023-10-06T01:41:07.461513+0000","last_undegraded":"2023-10-06T01:41:07.461513+0000","last_fullsized":"2023-10-06T01:41:07.461513+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:51:02.949231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.336635+0000","last_change":"2023-10-06T01:41:12.343382+0000","last_active":"2023-10-06T01:41:13.336635+0000","last_peered":"2023-10-06T01:41:13.336635+0000","last_clean":"2023-10-06T01:41:13.336635+0000","last_became_active":"2023-10-06T01:41:12.343088+0000","last_became_peered":"2023-10-06T01:41:12.343088+0000","last_unstale":"2023-10-06T01:41:13.336635+0000","last_undegraded":"2023-10-06T01:41:13.336635+0000","last_fullsized":"2023-10-06T01:41:13.336635+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:44:53.014481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.350157+0000","last_change":"2023-10-06T01:41:13.346852+0000","last_active":"2023-10-06T01:41:13.350157+0000","last_peered":"2023-10-06T01:41:13.350157+0000","last_clean":"2023-10-06T01:41:13.350157+0000","last_became_active":"2023-10-06T01:41:13.346571+0000","last_became_peered":"2023-10-06T01:41:13.346571+0000","last_unstale":"2023-10-06T01:41:13.350157+0000","last_undegraded":"2023-10-06T01:41:13.350157+0000","last_fullsized":"2023-10-06T01:41:13.350157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:25:04.563235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-06T01:41:07.461501+0000","last_change":"2023-10-06T01:41:07.456273+0000","last_active":"2023-10-06T01:41:07.461501+0000","last_peered":"2023-10-06T01:41:07.461501+0000","last_clean":"2023-10-06T01:41:07.461501+0000","last_became_active":"2023-10-06T01:41:07.456000+0000","last_became_peered":"2023-10-06T01:41:07.456000+0000","last_unstale":"2023-10-06T01:41:07.461501+0000","last_undegraded":"2023-10-06T01:41:07.461501+0000","last_fullsized":"2023-10-06T01:41:07.461501+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:01:42.633388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-10-06T01:41:12.349428+0000","last_change":"2023-10-06T01:41:12.342215+0000","last_active":"2023-10-06T01:41:12.349428+0000","last_peered":"2023-10-06T01:41:12.349428+0000","last_clean":"2023-10-06T01:41:12.349428+0000","last_became_active":"2023-10-06T01:41:12.341493+0000","last_became_peered":"2023-10-06T01:41:12.341493+0000","last_unstale":"2023-10-06T01:41:12.349428+0000","last_undegraded":"2023-10-06T01:41:12.349428+0000","last_fullsized":"2023-10-06T01:41:12.349428+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:49:54.658088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-06T01:41:07.461498+0000","last_change":"2023-10-06T01:41:07.456688+0000","last_active":"2023-10-06T01:41:07.461498+0000","last_peered":"2023-10-06T01:41:07.461498+0000","last_clean":"2023-10-06T01:41:07.461498+0000","last_became_active":"2023-10-06T01:41:07.456304+0000","last_became_peered":"2023-10-06T01:41:07.456304+0000","last_unstale":"2023-10-06T01:41:07.461498+0000","last_undegraded":"2023-10-06T01:41:07.461498+0000","last_fullsized":"2023-10-06T01:41:07.461498+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:11.306723+0000","last_change":"2023-10-06T01:41:11.306723+0000","last_active":"2023-10-06T01:41:11.306723+0000","last_peered":"2023-10-06T01:41:11.306723+0000","last_clean":"2023-10-06T01:41:11.306723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:11.306723+0000","last_undegraded":"2023-10-06T01:41:11.306723+0000","last_fullsized":"2023-10-06T01:41:11.306723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352905+0000","last_change":"2023-10-06T01:41:13.348241+0000","last_active":"2023-10-06T01:41:13.352905+0000","last_peered":"2023-10-06T01:41:13.352905+0000","last_clean":"2023-10-06T01:41:13.352905+0000","last_became_active":"2023-10-06T01:41:13.347979+0000","last_became_peered":"2023-10-06T01:41:13.347979+0000","last_unstale":"2023-10-06T01:41:13.352905+0000","last_undegraded":"2023-10-06T01:41:13.352905+0000","last_fullsized":"2023-10-06T01:41:13.352905+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:41:59.186469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-06T01:41:07.459128+0000","last_change":"2023-10-06T01:41:07.456388+0000","last_active":"2023-10-06T01:41:07.459128+0000","last_peered":"2023-10-06T01:41:07.459128+0000","last_clean":"2023-10-06T01:41:07.459128+0000","last_became_active":"2023-10-06T01:41:07.456045+0000","last_became_peered":"2023-10-06T01:41:07.456045+0000","last_unstale":"2023-10-06T01:41:07.459128+0000","last_undegraded":"2023-10-06T01:41:07.459128+0000","last_fullsized":"2023-10-06T01:41:07.459128+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.338387+0000","last_change":"2023-10-06T01:41:12.344268+0000","last_active":"2023-10-06T01:41:13.338387+0000","last_peered":"2023-10-06T01:41:13.338387+0000","last_clean":"2023-10-06T01:41:13.338387+0000","last_became_active":"2023-10-06T01:41:12.343996+0000","last_became_peered":"2023-10-06T01:41:12.343996+0000","last_unstale":"2023-10-06T01:41:13.338387+0000","last_undegraded":"2023-10-06T01:41:13.338387+0000","last_fullsized":"2023-10-06T01:41:13.338387+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:37:52.143727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.346184+0000","last_change":"2023-10-06T01:41:13.346184+0000","last_active":"2023-10-06T01:41:13.346184+0000","last_peered":"2023-10-06T01:41:13.346184+0000","last_clean":"2023-10-06T01:41:13.346184+0000","last_became_active":"2023-10-06T01:41:13.345570+0000","last_became_peered":"2023-10-06T01:41:13.345570+0000","last_unstale":"2023-10-06T01:41:13.346184+0000","last_undegraded":"2023-10-06T01:41:13.346184+0000","last_fullsized":"2023-10-06T01:41:13.346184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:42:46.214805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-10-06T01:41:07.259421+0000","last_change":"2023-10-06T01:41:07.258162+0000","last_active":"2023-10-06T01:41:07.259421+0000","last_peered":"2023-10-06T01:41:07.259421+0000","last_clean":"2023-10-06T01:41:07.259421+0000","last_became_active":"2023-10-06T01:41:07.257795+0000","last_became_peered":"2023-10-06T01:41:07.257795+0000","last_unstale":"2023-10-06T01:41:07.259421+0000","last_undegraded":"2023-10-06T01:41:07.259421+0000","last_fullsized":"2023-10-06T01:41:07.259421+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:05:22.934401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:11.306723+0000","last_change":"2023-10-06T01:41:11.306723+0000","last_active":"2023-10-06T01:41:11.306723+0000","last_peered":"2023-10-06T01:41:11.306723+0000","last_clean":"2023-10-06T01:41:11.306723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:11.306723+0000","last_undegraded":"2023-10-06T01:41:11.306723+0000","last_fullsized":"2023-10-06T01:41:11.306723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.349056+0000","last_change":"2023-10-06T01:41:13.344589+0000","last_active":"2023-10-06T01:41:13.349056+0000","last_peered":"2023-10-06T01:41:13.349056+0000","last_clean":"2023-10-06T01:41:13.349056+0000","last_became_active":"2023-10-06T01:41:13.344281+0000","last_became_peered":"2023-10-06T01:41:13.344281+0000","last_unstale":"2023-10-06T01:41:13.349056+0000","last_undegraded":"2023-10-06T01:41:13.349056+0000","last_fullsized":"2023-10-06T01:41:13.349056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:32:17.938309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"19'1","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.329842+0000","last_change":"2023-10-06T01:41:12.347655+0000","last_active":"2023-10-06T01:41:13.329842+0000","last_peered":"2023-10-06T01:41:13.329842+0000","last_clean":"2023-10-06T01:41:13.329842+0000","last_became_active":"2023-10-06T01:41:12.347353+0000","last_became_peered":"2023-10-06T01:41:12.347353+0000","last_unstale":"2023-10-06T01:41:13.329842+0000","last_undegraded":"2023-10-06T01:41:13.329842+0000","last_fullsized":"2023-10-06T01:41:13.329842+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:49:54.658088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:11.306723+0000","last_change":"2023-10-06T01:41:11.306723+0000","last_active":"2023-10-06T01:41:11.306723+0000","last_peered":"2023-10-06T01:41:11.306723+0000","last_clean":"2023-10-06T01:41:11.306723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:11.306723+0000","last_undegraded":"2023-10-06T01:41:11.306723+0000","last_fullsized":"2023-10-06T01:41:11.306723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:11.306723+0000","last_change":"2023-10-06T01:41:11.306723+0000","last_active":"2023-10-06T01:41:11.306723+0000","last_peered":"2023-10-06T01:41:11.306723+0000","last_clean":"2023-10-06T01:41:11.306723+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:11.306723+0000","last_undegraded":"2023-10-06T01:41:11.306723+0000","last_fullsized":"2023-10-06T01:41:11.306723+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"3.1e","version":"19'1","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.337057+0000","last_change":"2023-10-06T01:41:12.340776+0000","last_active":"2023-10-06T01:41:13.337057+0000","last_peered":"2023-10-06T01:41:13.337057+0000","last_clean":"2023-10-06T01:41:13.337057+0000","last_became_active":"2023-10-06T01:41:12.340294+0000","last_became_peered":"2023-10-06T01:41:12.340294+0000","last_unstale":"2023-10-06T01:41:13.337057+0000","last_undegraded":"2023-10-06T01:41:13.337057+0000","last_fullsized":"2023-10-06T01:41:13.337057+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:37:13.672716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.355273+0000","last_change":"2023-10-06T01:41:13.355273+0000","last_active":"2023-10-06T01:41:13.355273+0000","last_peered":"2023-10-06T01:41:13.355273+0000","last_clean":"2023-10-06T01:41:13.355273+0000","last_became_active":"2023-10-06T01:41:13.354977+0000","last_became_peered":"2023-10-06T01:41:13.354977+0000","last_unstale":"2023-10-06T01:41:13.355273+0000","last_undegraded":"2023-10-06T01:41:13.355273+0000","last_fullsized":"2023-10-06T01:41:13.355273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:21:42.851681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"19'4","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.345488+0000","last_change":"2023-10-06T01:41:12.338431+0000","last_active":"2023-10-06T01:41:13.345488+0000","last_peered":"2023-10-06T01:41:13.345488+0000","last_clean":"2023-10-06T01:41:13.345488+0000","last_became_active":"2023-10-06T01:41:12.338081+0000","last_became_peered":"2023-10-06T01:41:12.338081+0000","last_unstale":"2023-10-06T01:41:13.345488+0000","last_undegraded":"2023-10-06T01:41:13.345488+0000","last_fullsized":"2023-10-06T01:41:13.345488+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:16:14.826869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.355216+0000","last_change":"2023-10-06T01:41:13.355216+0000","last_active":"2023-10-06T01:41:13.355216+0000","last_peered":"2023-10-06T01:41:13.355216+0000","last_clean":"2023-10-06T01:41:13.355216+0000","last_became_active":"2023-10-06T01:41:13.354853+0000","last_became_peered":"2023-10-06T01:41:13.354853+0000","last_unstale":"2023-10-06T01:41:13.355216+0000","last_undegraded":"2023-10-06T01:41:13.355216+0000","last_fullsized":"2023-10-06T01:41:13.355216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:42:44.799267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:41:12.313668+0000","last_change":"2023-10-06T01:41:12.313668+0000","last_active":"2023-10-06T01:41:12.313668+0000","last_peered":"2023-10-06T01:41:12.313668+0000","last_clean":"2023-10-06T01:41:12.313668+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:41:12.313668+0000","last_undegraded":"2023-10-06T01:41:12.313668+0000","last_fullsized":"2023-10-06T01:41:12.313668+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.353822+0000","last_change":"2023-10-06T01:41:13.353822+0000","last_active":"2023-10-06T01:41:13.353822+0000","last_peered":"2023-10-06T01:41:13.353822+0000","last_clean":"2023-10-06T01:41:13.353822+0000","last_became_active":"2023-10-06T01:41:13.353292+0000","last_became_peered":"2023-10-06T01:41:13.353292+0000","last_unstale":"2023-10-06T01:41:13.353822+0000","last_undegraded":"2023-10-06T01:41:13.353822+0000","last_fullsized":"2023-10-06T01:41:13.353822+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:52:48.286318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356636+0000","last_change":"2023-10-06T01:41:13.353662+0000","last_active":"2023-10-06T01:41:13.356636+0000","last_peered":"2023-10-06T01:41:13.356636+0000","last_clean":"2023-10-06T01:41:13.356636+0000","last_became_active":"2023-10-06T01:41:13.353269+0000","last_became_peered":"2023-10-06T01:41:13.353269+0000","last_unstale":"2023-10-06T01:41:13.356636+0000","last_undegraded":"2023-10-06T01:41:13.356636+0000","last_fullsized":"2023-10-06T01:41:13.356636+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:11:44.715827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":54,"acting":54,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":554,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":30,"acting":30,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298584,"kb_used_data":732,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073256,"statfs":{"total":96636764160,"available":96331014144,"internally_reserved":0,"allocated":749568,"data_stored":638029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.022}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97599999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297436,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074404,"statfs":{"total":96636764160,"available":96332189696,"internally_reserved":0,"allocated":151552,"data_stored":46308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58599999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70299999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298068,"kb_used_data":780,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073772,"statfs":{"total":96636764160,"available":96331542528,"internally_reserved":0,"allocated":798720,"data_stored":664552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297476,"kb_used_data":188,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074364,"statfs":{"total":96636764160,"available":96332148736,"internally_reserved":0,"allocated":192512,"data_stored":66744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72899999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297440,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074400,"statfs":{"total":96636764160,"available":96332185600,"internally_reserved":0,"allocated":155648,"data_stored":47661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28399999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297476,"kb_used_data":188,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074364,"statfs":{"total":96636764160,"available":96332148736,"internally_reserved":0,"allocated":192512,"data_stored":66280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297440,"kb_used_data":152,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074400,"statfs":{"total":96636764160,"available":96332185600,"internally_reserved":0,"allocated":155648,"data_stored":47661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54900000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574851,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297436,"kb_used_data":148,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074404,"statfs":{"total":96636764160,"available":96332189696,"internally_reserved":0,"allocated":151552,"data_stored":46308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-06T01:41:17.834 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-06T01:41:17.985 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.985+0000 7f743c646700 1 Processor -- start 2023-10-06T01:41:17.985 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.985+0000 7f743c646700 1 -- start start 2023-10-06T01:41:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.986+0000 7f743c646700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f7434159510 0x7f74340acb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.986+0000 7f743c646700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f74340685d0 0x7f74341633d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.986+0000 7f743c646700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74340adbb0 con 0x7f74340ad070 2023-10-06T01:41:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.986+0000 7f743c646700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74340ae210 con 0x7f7434159510 2023-10-06T01:41:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.986+0000 7f743c646700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74340ad5f0 con 0x7f74340685d0 2023-10-06T01:41:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.986+0000 7f743abe3700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f74340685d0 0x7f74341633d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:17.988 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.986+0000 7f743a3e2700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f7434159510 0x7f74340acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:17.989 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.986+0000 7f743a3e2700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f7434159510 0x7f74340acb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.27:42988/0 (socket says 172.21.15.27:42988) 2023-10-06T01:41:17.989 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.986+0000 7f743a3e2700 1 -- 172.21.15.27:0/377109965 learned_addr learned my addr 172.21.15.27:0/377109965 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:17.989 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.986+0000 7f743abe3700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f74340685d0 0x7f74341633d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.27:35808/0 (socket says 172.21.15.27:35808) 2023-10-06T01:41:17.989 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.986+0000 7f7439be1700 1 --1- 172.21.15.27:0/377109965 >> v1:172.21.15.124:6789/0 conn(0x7f74340ad070 0x7f7434067ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.27:46092/0 (socket says 172.21.15.27:46092) 2023-10-06T01:41:17.989 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.987+0000 7f743a3e2700 1 -- 172.21.15.27:0/377109965 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f74340685d0 msgr2=0x7f74341633d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:17.989 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.987+0000 7f743a3e2700 1 --2- 172.21.15.27:0/377109965 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f74340685d0 0x7f74341633d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:17.989 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.987+0000 7f743a3e2700 1 -- 172.21.15.27:0/377109965 >> v1:172.21.15.124:6789/0 conn(0x7f74340ad070 legacy=0x7f7434067ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:17.989 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.987+0000 7f743a3e2700 1 -- 172.21.15.27:0/377109965 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74340ad8d0 con 0x7f7434159510 2023-10-06T01:41:17.989 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.987+0000 7f74393e0700 1 -- 172.21.15.27:0/377109965 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2299649737 0 0) 0x7f74340adbb0 con 0x7f74340ad070 2023-10-06T01:41:17.989 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.987+0000 7f743abe3700 1 --2- 172.21.15.27:0/377109965 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f74340685d0 0x7f74341633d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:41:17.990 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.987+0000 7f743a3e2700 1 --2- 172.21.15.27:0/377109965 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7434159510 0x7f74340acb30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f742c0099b0 tx=0x7f742c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbaf22599f376dd9 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:17.990 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.987+0000 7f74393e0700 1 -- 172.21.15.27:0/377109965 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f742c009ea0 con 0x7f7434159510 2023-10-06T01:41:17.990 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.987+0000 7f74393e0700 1 -- 172.21.15.27:0/377109965 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f742c00b860 con 0x7f7434159510 2023-10-06T01:41:17.990 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.987+0000 7f74393e0700 1 -- 172.21.15.27:0/377109965 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f742c00bc00 con 0x7f7434159510 2023-10-06T01:41:17.990 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.988+0000 7f743c646700 1 -- 172.21.15.27:0/377109965 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7434159510 msgr2=0x7f74340acb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:17.990 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.988+0000 7f743c646700 1 --2- 172.21.15.27:0/377109965 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7434159510 0x7f74340acb30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f742c0099b0 tx=0x7f742c009250 comp rx=0 tx=0).stop 2023-10-06T01:41:17.990 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.988+0000 7f743c646700 1 -- 172.21.15.27:0/377109965 shutdown_connections 2023-10-06T01:41:17.990 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.988+0000 7f743c646700 1 --2- 172.21.15.27:0/377109965 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f74340685d0 0x7f74341633d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:17.990 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.988+0000 7f743c646700 1 --2- 172.21.15.27:0/377109965 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7434159510 0x7f74340acb30 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:17.990 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.988+0000 7f743c646700 1 -- 172.21.15.27:0/377109965 >> 172.21.15.27:0/377109965 conn(0x7f743406e6b0 msgr2=0x7f743406afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:17.991 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.988+0000 7f743c646700 1 -- 172.21.15.27:0/377109965 shutdown_connections 2023-10-06T01:41:17.991 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.988+0000 7f743c646700 1 -- 172.21.15.27:0/377109965 wait complete. 2023-10-06T01:41:17.993 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.989+0000 7f743c646700 1 Processor -- start 2023-10-06T01:41:17.993 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.989+0000 7f743c646700 1 -- start start 2023-10-06T01:41:17.993 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.989+0000 7f743c646700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74340685d0 0x7f74340c28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:17.993 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.989+0000 7f743c646700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f74340ad070 0x7f743415ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:17.993 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.989+0000 7f743c646700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7434159510 0x7f743415b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:17.993 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.989+0000 7f743c646700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7434143a40 con 0x7f74340685d0 2023-10-06T01:41:17.994 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.989+0000 7f743c646700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74340ab1f0 con 0x7f74340ad070 2023-10-06T01:41:17.994 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.989+0000 7f743c646700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74340aa5e0 con 0x7f7434159510 2023-10-06T01:41:17.994 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.990+0000 7f743a3e2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74340685d0 0x7f74340c28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:17.994 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.990+0000 7f743a3e2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74340685d0 0x7f74340c28b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55788/0 (socket says 172.21.15.27:55788) 2023-10-06T01:41:17.994 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.990+0000 7f743a3e2700 1 -- 172.21.15.27:0/157090084 learned_addr learned my addr 172.21.15.27:0/157090084 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:17.994 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.990+0000 7f743abe3700 1 --2- 172.21.15.27:0/157090084 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7434159510 0x7f743415b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:17.994 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.990+0000 7f7439be1700 1 --2- 172.21.15.27:0/157090084 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f74340ad070 0x7f743415ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:17.994 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.990+0000 7f743a3e2700 1 -- 172.21.15.27:0/157090084 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7434159510 msgr2=0x7f743415b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:17.994 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.990+0000 7f743a3e2700 1 --2- 172.21.15.27:0/157090084 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7434159510 0x7f743415b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:17.994 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.990+0000 7f743a3e2700 1 -- 172.21.15.27:0/157090084 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f74340ad070 msgr2=0x7f743415ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:17.994 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.990+0000 7f743a3e2700 1 --2- 172.21.15.27:0/157090084 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f74340ad070 0x7f743415ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:17.995 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.990+0000 7f743a3e2700 1 -- 172.21.15.27:0/157090084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7424009a20 con 0x7f74340685d0 2023-10-06T01:41:17.995 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.990+0000 7f743a3e2700 1 --2- 172.21.15.27:0/157090084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74340685d0 0x7f74340c28b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f742c00b3c0 tx=0x7f742c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:17.995 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.990+0000 7f74327fc700 1 -- 172.21.15.27:0/157090084 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f742c000d10 con 0x7f74340685d0 2023-10-06T01:41:17.995 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.990+0000 7f74327fc700 1 -- 172.21.15.27:0/157090084 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f742c00b9d0 con 0x7f74340685d0 2023-10-06T01:41:17.995 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.990+0000 7f743c646700 1 -- 172.21.15.27:0/157090084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f742c0094f0 con 0x7f74340685d0 2023-10-06T01:41:17.995 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.990+0000 7f74327fc700 1 -- 172.21.15.27:0/157090084 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f742c025420 con 0x7f74340685d0 2023-10-06T01:41:17.995 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.990+0000 7f743c646700 1 -- 172.21.15.27:0/157090084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74340aaba0 con 0x7f74340685d0 2023-10-06T01:41:17.995 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.991+0000 7f743c646700 1 -- 172.21.15.27:0/157090084 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7434064050 con 0x7f74340685d0 2023-10-06T01:41:17.995 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.992+0000 7f74327fc700 1 -- 172.21.15.27:0/157090084 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f742c023070 con 0x7f74340685d0 2023-10-06T01:41:17.995 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.992+0000 7f74327fc700 1 --2- 172.21.15.27:0/157090084 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7420066f30 0x7f74200693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:17.995 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.992+0000 7f74327fc700 1 -- 172.21.15.27:0/157090084 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f742c085b40 con 0x7f74340685d0 2023-10-06T01:41:17.995 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.992+0000 7f7439be1700 1 --2- 172.21.15.27:0/157090084 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7420066f30 0x7f74200693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:17.996 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.993+0000 7f7439be1700 1 --2- 172.21.15.27:0/157090084 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7420066f30 0x7f74200693e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7434154820 tx=0x7f7428007d50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:17.996 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:17.996+0000 7f74327fc700 1 -- 172.21.15.27:0/157090084 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f742c052d10 con 0x7f74340685d0 2023-10-06T01:41:18.185 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.185+0000 7f743c646700 1 -- 172.21.15.27:0/157090084 --> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f74340ab330 con 0x7f7420066f30 2023-10-06T01:41:18.193 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.193+0000 7f74327fc700 1 -- 172.21.15.27:0/157090084 <== mgr.4100 v2:172.21.15.124:6832/90278 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286886 (secure 0 0 0) 0x7f7428004ca0 con 0x7f7420066f30 2023-10-06T01:41:18.196 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:18.198 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.196+0000 7f743c646700 1 -- 172.21.15.27:0/157090084 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7420066f30 msgr2=0x7f74200693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:18.199 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.196+0000 7f743c646700 1 --2- 172.21.15.27:0/157090084 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7420066f30 0x7f74200693e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7434154820 tx=0x7f7428007d50 comp rx=0 tx=0).stop 2023-10-06T01:41:18.199 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.196+0000 7f743c646700 1 -- 172.21.15.27:0/157090084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74340685d0 msgr2=0x7f74340c28b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:18.199 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.196+0000 7f743c646700 1 --2- 172.21.15.27:0/157090084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74340685d0 0x7f74340c28b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f742c00b3c0 tx=0x7f742c017000 comp rx=0 tx=0).stop 2023-10-06T01:41:18.199 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.196+0000 7f743c646700 1 -- 172.21.15.27:0/157090084 shutdown_connections 2023-10-06T01:41:18.199 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.196+0000 7f743c646700 1 --2- 172.21.15.27:0/157090084 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7420066f30 0x7f74200693e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:18.199 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.196+0000 7f743c646700 1 --2- 172.21.15.27:0/157090084 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7434159510 0x7f743415b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:18.199 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.196+0000 7f743c646700 1 --2- 172.21.15.27:0/157090084 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f74340685d0 0x7f74340c28b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:18.199 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.196+0000 7f743c646700 1 --2- 172.21.15.27:0/157090084 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f74340ad070 0x7f743415ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:18.199 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.196+0000 7f743c646700 1 -- 172.21.15.27:0/157090084 >> 172.21.15.27:0/157090084 conn(0x7f743406e6b0 msgr2=0x7f7434154ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:18.199 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.196+0000 7f743c646700 1 -- 172.21.15.27:0/157090084 shutdown_connections 2023-10-06T01:41:18.199 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.196+0000 7f743c646700 1 -- 172.21.15.27:0/157090084 wait complete. 2023-10-06T01:41:18.200 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-06T01:41:18.208 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2023-10-06T01:41:18.039083+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":112,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2381088,"kb_used_data":2784,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752593632,"statfs":{"total":773094113280,"available":770655879168,"internally_reserved":0,"allocated":2850816,"data_stored":1760126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001420"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.349688+0000","last_change":"2023-10-06T01:41:13.346440+0000","last_active":"2023-10-06T01:41:13.349688+0000","last_peered":"2023-10-06T01:41:13.349688+0000","last_clean":"2023-10-06T01:41:13.349688+0000","last_became_active":"2023-10-06T01:41:13.343345+0000","last_became_peered":"2023-10-06T01:41:13.343345+0000","last_unstale":"2023-10-06T01:41:13.349688+0000","last_undegraded":"2023-10-06T01:41:13.349688+0000","last_fullsized":"2023-10-06T01:41:13.349688+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:15:41.848935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354765+0000","last_change":"2023-10-06T01:41:13.348847+0000","last_active":"2023-10-06T01:41:13.354765+0000","last_peered":"2023-10-06T01:41:13.354765+0000","last_clean":"2023-10-06T01:41:13.354765+0000","last_became_active":"2023-10-06T01:41:13.348585+0000","last_became_peered":"2023-10-06T01:41:13.348585+0000","last_unstale":"2023-10-06T01:41:13.354765+0000","last_undegraded":"2023-10-06T01:41:13.354765+0000","last_fullsized":"2023-10-06T01:41:13.354765+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:31:20.840764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354463+0000","last_change":"2023-10-06T01:41:13.345122+0000","last_active":"2023-10-06T01:41:13.354463+0000","last_peered":"2023-10-06T01:41:13.354463+0000","last_clean":"2023-10-06T01:41:13.354463+0000","last_became_active":"2023-10-06T01:41:13.344752+0000","last_became_peered":"2023-10-06T01:41:13.344752+0000","last_unstale":"2023-10-06T01:41:13.354463+0000","last_undegraded":"2023-10-06T01:41:13.354463+0000","last_fullsized":"2023-10-06T01:41:13.354463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:42:56.041827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.342734+0000","last_change":"2023-10-06T01:41:13.340589+0000","last_active":"2023-10-06T01:41:13.342734+0000","last_peered":"2023-10-06T01:41:13.342734+0000","last_clean":"2023-10-06T01:41:13.342734+0000","last_became_active":"2023-10-06T01:41:13.340339+0000","last_became_peered":"2023-10-06T01:41:13.340339+0000","last_unstale":"2023-10-06T01:41:13.342734+0000","last_undegraded":"2023-10-06T01:41:13.342734+0000","last_fullsized":"2023-10-06T01:41:13.342734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:32:17.938309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.342002+0000","last_change":"2023-10-06T01:41:13.342002+0000","last_active":"2023-10-06T01:41:13.342002+0000","last_peered":"2023-10-06T01:41:13.342002+0000","last_clean":"2023-10-06T01:41:13.342002+0000","last_became_active":"2023-10-06T01:41:13.341403+0000","last_became_peered":"2023-10-06T01:41:13.341403+0000","last_unstale":"2023-10-06T01:41:13.342002+0000","last_undegraded":"2023-10-06T01:41:13.342002+0000","last_fullsized":"2023-10-06T01:41:13.342002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:34:55.375989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354785+0000","last_change":"2023-10-06T01:41:13.345138+0000","last_active":"2023-10-06T01:41:13.354785+0000","last_peered":"2023-10-06T01:41:13.354785+0000","last_clean":"2023-10-06T01:41:13.354785+0000","last_became_active":"2023-10-06T01:41:13.344810+0000","last_became_peered":"2023-10-06T01:41:13.344810+0000","last_unstale":"2023-10-06T01:41:13.354785+0000","last_undegraded":"2023-10-06T01:41:13.354785+0000","last_fullsized":"2023-10-06T01:41:13.354785+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:17:59.027125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356675+0000","last_change":"2023-10-06T01:41:13.352141+0000","last_active":"2023-10-06T01:41:13.356675+0000","last_peered":"2023-10-06T01:41:13.356675+0000","last_clean":"2023-10-06T01:41:13.356675+0000","last_became_active":"2023-10-06T01:41:13.351628+0000","last_became_peered":"2023-10-06T01:41:13.351628+0000","last_unstale":"2023-10-06T01:41:13.356675+0000","last_undegraded":"2023-10-06T01:41:13.356675+0000","last_fullsized":"2023-10-06T01:41:13.356675+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:03:37.385158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354278+0000","last_change":"2023-10-06T01:41:13.345960+0000","last_active":"2023-10-06T01:41:13.354278+0000","last_peered":"2023-10-06T01:41:13.354278+0000","last_clean":"2023-10-06T01:41:13.354278+0000","last_became_active":"2023-10-06T01:41:13.345702+0000","last_became_peered":"2023-10-06T01:41:13.345702+0000","last_unstale":"2023-10-06T01:41:13.354278+0000","last_undegraded":"2023-10-06T01:41:13.354278+0000","last_fullsized":"2023-10-06T01:41:13.354278+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:03:37.385158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.343792+0000","last_change":"2023-10-06T01:41:13.343792+0000","last_active":"2023-10-06T01:41:13.343792+0000","last_peered":"2023-10-06T01:41:13.343792+0000","last_clean":"2023-10-06T01:41:13.343792+0000","last_became_active":"2023-10-06T01:41:13.343432+0000","last_became_peered":"2023-10-06T01:41:13.343432+0000","last_unstale":"2023-10-06T01:41:13.343792+0000","last_undegraded":"2023-10-06T01:41:13.343792+0000","last_fullsized":"2023-10-06T01:41:13.343792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:52:48.286318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354747+0000","last_change":"2023-10-06T01:41:13.347399+0000","last_active":"2023-10-06T01:41:13.354747+0000","last_peered":"2023-10-06T01:41:13.354747+0000","last_clean":"2023-10-06T01:41:13.354747+0000","last_became_active":"2023-10-06T01:41:13.347136+0000","last_became_peered":"2023-10-06T01:41:13.347136+0000","last_unstale":"2023-10-06T01:41:13.354747+0000","last_undegraded":"2023-10-06T01:41:13.354747+0000","last_fullsized":"2023-10-06T01:41:13.354747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:42:23.199959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354772+0000","last_change":"2023-10-06T01:41:13.350059+0000","last_active":"2023-10-06T01:41:13.354772+0000","last_peered":"2023-10-06T01:41:13.354772+0000","last_clean":"2023-10-06T01:41:13.354772+0000","last_became_active":"2023-10-06T01:41:13.349673+0000","last_became_peered":"2023-10-06T01:41:13.349673+0000","last_unstale":"2023-10-06T01:41:13.354772+0000","last_undegraded":"2023-10-06T01:41:13.354772+0000","last_fullsized":"2023-10-06T01:41:13.354772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:12:57.315612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351538+0000","last_change":"2023-10-06T01:41:13.348001+0000","last_active":"2023-10-06T01:41:13.351538+0000","last_peered":"2023-10-06T01:41:13.351538+0000","last_clean":"2023-10-06T01:41:13.351538+0000","last_became_active":"2023-10-06T01:41:13.347719+0000","last_became_peered":"2023-10-06T01:41:13.347719+0000","last_unstale":"2023-10-06T01:41:13.351538+0000","last_undegraded":"2023-10-06T01:41:13.351538+0000","last_fullsized":"2023-10-06T01:41:13.351538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:37:53.150671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.355181+0000","last_change":"2023-10-06T01:41:13.352875+0000","last_active":"2023-10-06T01:41:13.355181+0000","last_peered":"2023-10-06T01:41:13.355181+0000","last_clean":"2023-10-06T01:41:13.355181+0000","last_became_active":"2023-10-06T01:41:13.352589+0000","last_became_peered":"2023-10-06T01:41:13.352589+0000","last_unstale":"2023-10-06T01:41:13.355181+0000","last_undegraded":"2023-10-06T01:41:13.355181+0000","last_fullsized":"2023-10-06T01:41:13.355181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:49:55.665032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354387+0000","last_change":"2023-10-06T01:41:13.348362+0000","last_active":"2023-10-06T01:41:13.354387+0000","last_peered":"2023-10-06T01:41:13.354387+0000","last_clean":"2023-10-06T01:41:13.354387+0000","last_became_active":"2023-10-06T01:41:13.348181+0000","last_became_peered":"2023-10-06T01:41:13.348181+0000","last_unstale":"2023-10-06T01:41:13.354387+0000","last_undegraded":"2023-10-06T01:41:13.354387+0000","last_fullsized":"2023-10-06T01:41:13.354387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:27:54.074728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356697+0000","last_change":"2023-10-06T01:41:13.352048+0000","last_active":"2023-10-06T01:41:13.356697+0000","last_peered":"2023-10-06T01:41:13.356697+0000","last_clean":"2023-10-06T01:41:13.356697+0000","last_became_active":"2023-10-06T01:41:13.351640+0000","last_became_peered":"2023-10-06T01:41:13.351640+0000","last_unstale":"2023-10-06T01:41:13.356697+0000","last_undegraded":"2023-10-06T01:41:13.356697+0000","last_fullsized":"2023-10-06T01:41:13.356697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:42:23.199959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354602+0000","last_change":"2023-10-06T01:41:13.345677+0000","last_active":"2023-10-06T01:41:13.354602+0000","last_peered":"2023-10-06T01:41:13.354602+0000","last_clean":"2023-10-06T01:41:13.354602+0000","last_became_active":"2023-10-06T01:41:13.345257+0000","last_became_peered":"2023-10-06T01:41:13.345257+0000","last_unstale":"2023-10-06T01:41:13.354602+0000","last_undegraded":"2023-10-06T01:41:13.354602+0000","last_fullsized":"2023-10-06T01:41:13.354602+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:27:07.040029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.344490+0000","last_change":"2023-10-06T01:41:13.344490+0000","last_active":"2023-10-06T01:41:13.344490+0000","last_peered":"2023-10-06T01:41:13.344490+0000","last_clean":"2023-10-06T01:41:13.344490+0000","last_became_active":"2023-10-06T01:41:13.344220+0000","last_became_peered":"2023-10-06T01:41:13.344220+0000","last_unstale":"2023-10-06T01:41:13.344490+0000","last_undegraded":"2023-10-06T01:41:13.344490+0000","last_fullsized":"2023-10-06T01:41:13.344490+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T01:51:57.229301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354756+0000","last_change":"2023-10-06T01:41:13.350575+0000","last_active":"2023-10-06T01:41:13.354756+0000","last_peered":"2023-10-06T01:41:13.354756+0000","last_clean":"2023-10-06T01:41:13.354756+0000","last_became_active":"2023-10-06T01:41:13.350311+0000","last_became_peered":"2023-10-06T01:41:13.350311+0000","last_unstale":"2023-10-06T01:41:13.354756+0000","last_undegraded":"2023-10-06T01:41:13.354756+0000","last_fullsized":"2023-10-06T01:41:13.354756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:39:03.091509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351594+0000","last_change":"2023-10-06T01:41:13.347386+0000","last_active":"2023-10-06T01:41:13.351594+0000","last_peered":"2023-10-06T01:41:13.351594+0000","last_clean":"2023-10-06T01:41:13.351594+0000","last_became_active":"2023-10-06T01:41:13.347101+0000","last_became_peered":"2023-10-06T01:41:13.347101+0000","last_unstale":"2023-10-06T01:41:13.351594+0000","last_undegraded":"2023-10-06T01:41:13.351594+0000","last_fullsized":"2023-10-06T01:41:13.351594+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:13:15.011521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351504+0000","last_change":"2023-10-06T01:41:13.342640+0000","last_active":"2023-10-06T01:41:13.351504+0000","last_peered":"2023-10-06T01:41:13.351504+0000","last_clean":"2023-10-06T01:41:13.351504+0000","last_became_active":"2023-10-06T01:41:13.342157+0000","last_became_peered":"2023-10-06T01:41:13.342157+0000","last_unstale":"2023-10-06T01:41:13.351504+0000","last_undegraded":"2023-10-06T01:41:13.351504+0000","last_fullsized":"2023-10-06T01:41:13.351504+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:07:52.521701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356729+0000","last_change":"2023-10-06T01:41:13.352582+0000","last_active":"2023-10-06T01:41:13.356729+0000","last_peered":"2023-10-06T01:41:13.356729+0000","last_clean":"2023-10-06T01:41:13.356729+0000","last_became_active":"2023-10-06T01:41:13.352281+0000","last_became_peered":"2023-10-06T01:41:13.352281+0000","last_unstale":"2023-10-06T01:41:13.356729+0000","last_undegraded":"2023-10-06T01:41:13.356729+0000","last_fullsized":"2023-10-06T01:41:13.356729+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:31:20.840764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.345324+0000","last_change":"2023-10-06T01:41:13.345324+0000","last_active":"2023-10-06T01:41:13.345324+0000","last_peered":"2023-10-06T01:41:13.345324+0000","last_clean":"2023-10-06T01:41:13.345324+0000","last_became_active":"2023-10-06T01:41:13.344997+0000","last_became_peered":"2023-10-06T01:41:13.344997+0000","last_unstale":"2023-10-06T01:41:13.345324+0000","last_undegraded":"2023-10-06T01:41:13.345324+0000","last_fullsized":"2023-10-06T01:41:13.345324+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:27:54.074728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.355108+0000","last_change":"2023-10-06T01:41:13.355108+0000","last_active":"2023-10-06T01:41:13.355108+0000","last_peered":"2023-10-06T01:41:13.355108+0000","last_clean":"2023-10-06T01:41:13.355108+0000","last_became_active":"2023-10-06T01:41:13.354621+0000","last_became_peered":"2023-10-06T01:41:13.354621+0000","last_unstale":"2023-10-06T01:41:13.355108+0000","last_undegraded":"2023-10-06T01:41:13.355108+0000","last_fullsized":"2023-10-06T01:41:13.355108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:34:55.375989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356702+0000","last_change":"2023-10-06T01:41:13.352301+0000","last_active":"2023-10-06T01:41:13.356702+0000","last_peered":"2023-10-06T01:41:13.356702+0000","last_clean":"2023-10-06T01:41:13.356702+0000","last_became_active":"2023-10-06T01:41:13.351858+0000","last_became_peered":"2023-10-06T01:41:13.351858+0000","last_unstale":"2023-10-06T01:41:13.356702+0000","last_undegraded":"2023-10-06T01:41:13.356702+0000","last_fullsized":"2023-10-06T01:41:13.356702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:42:46.214805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.348956+0000","last_change":"2023-10-06T01:41:13.344094+0000","last_active":"2023-10-06T01:41:13.348956+0000","last_peered":"2023-10-06T01:41:13.348956+0000","last_clean":"2023-10-06T01:41:13.348956+0000","last_became_active":"2023-10-06T01:41:13.343796+0000","last_became_peered":"2023-10-06T01:41:13.343796+0000","last_unstale":"2023-10-06T01:41:13.348956+0000","last_undegraded":"2023-10-06T01:41:13.348956+0000","last_fullsized":"2023-10-06T01:41:13.348956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:21:30.451862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.355700+0000","last_change":"2023-10-06T01:41:13.350352+0000","last_active":"2023-10-06T01:41:13.355700+0000","last_peered":"2023-10-06T01:41:13.355700+0000","last_clean":"2023-10-06T01:41:13.355700+0000","last_became_active":"2023-10-06T01:41:13.350133+0000","last_became_peered":"2023-10-06T01:41:13.350133+0000","last_unstale":"2023-10-06T01:41:13.355700+0000","last_undegraded":"2023-10-06T01:41:13.355700+0000","last_fullsized":"2023-10-06T01:41:13.355700+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:05:37.961571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354695+0000","last_change":"2023-10-06T01:41:13.353241+0000","last_active":"2023-10-06T01:41:13.354695+0000","last_peered":"2023-10-06T01:41:13.354695+0000","last_clean":"2023-10-06T01:41:13.354695+0000","last_became_active":"2023-10-06T01:41:13.352824+0000","last_became_peered":"2023-10-06T01:41:13.352824+0000","last_unstale":"2023-10-06T01:41:13.354695+0000","last_undegraded":"2023-10-06T01:41:13.354695+0000","last_fullsized":"2023-10-06T01:41:13.354695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:52:48.286318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"19'1","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.348819+0000","last_change":"2023-10-06T01:41:12.339675+0000","last_active":"2023-10-06T01:41:13.348819+0000","last_peered":"2023-10-06T01:41:13.348819+0000","last_clean":"2023-10-06T01:41:13.348819+0000","last_became_active":"2023-10-06T01:41:12.339258+0000","last_became_peered":"2023-10-06T01:41:12.339258+0000","last_unstale":"2023-10-06T01:41:13.348819+0000","last_undegraded":"2023-10-06T01:41:13.348819+0000","last_fullsized":"2023-10-06T01:41:13.348819+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:49:54.658088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352444+0000","last_change":"2023-10-06T01:41:13.352444+0000","last_active":"2023-10-06T01:41:13.352444+0000","last_peered":"2023-10-06T01:41:13.352444+0000","last_clean":"2023-10-06T01:41:13.352444+0000","last_became_active":"2023-10-06T01:41:13.352134+0000","last_became_peered":"2023-10-06T01:41:13.352134+0000","last_unstale":"2023-10-06T01:41:13.352444+0000","last_undegraded":"2023-10-06T01:41:13.352444+0000","last_fullsized":"2023-10-06T01:41:13.352444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:11:26.794025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.336930+0000","last_change":"2023-10-06T01:41:12.347654+0000","last_active":"2023-10-06T01:41:13.336930+0000","last_peered":"2023-10-06T01:41:13.336930+0000","last_clean":"2023-10-06T01:41:13.336930+0000","last_became_active":"2023-10-06T01:41:12.347386+0000","last_became_peered":"2023-10-06T01:41:12.347386+0000","last_unstale":"2023-10-06T01:41:13.336930+0000","last_undegraded":"2023-10-06T01:41:13.336930+0000","last_fullsized":"2023-10-06T01:41:13.336930+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:07:30.529679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352350+0000","last_change":"2023-10-06T01:41:13.352350+0000","last_active":"2023-10-06T01:41:13.352350+0000","last_peered":"2023-10-06T01:41:13.352350+0000","last_clean":"2023-10-06T01:41:13.352350+0000","last_became_active":"2023-10-06T01:41:13.352091+0000","last_became_peered":"2023-10-06T01:41:13.352091+0000","last_unstale":"2023-10-06T01:41:13.352350+0000","last_undegraded":"2023-10-06T01:41:13.352350+0000","last_fullsized":"2023-10-06T01:41:13.352350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:31:20.840764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-10-06T01:41:12.349520+0000","last_change":"2023-10-06T01:41:12.343086+0000","last_active":"2023-10-06T01:41:12.349520+0000","last_peered":"2023-10-06T01:41:12.349520+0000","last_clean":"2023-10-06T01:41:12.349520+0000","last_became_active":"2023-10-06T01:41:12.342595+0000","last_became_peered":"2023-10-06T01:41:12.342595+0000","last_unstale":"2023-10-06T01:41:12.349520+0000","last_undegraded":"2023-10-06T01:41:12.349520+0000","last_fullsized":"2023-10-06T01:41:12.349520+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:15:40.841990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351933+0000","last_change":"2023-10-06T01:41:13.347027+0000","last_active":"2023-10-06T01:41:13.351933+0000","last_peered":"2023-10-06T01:41:13.351933+0000","last_clean":"2023-10-06T01:41:13.351933+0000","last_became_active":"2023-10-06T01:41:13.346469+0000","last_became_peered":"2023-10-06T01:41:13.346469+0000","last_unstale":"2023-10-06T01:41:13.351933+0000","last_undegraded":"2023-10-06T01:41:13.351933+0000","last_fullsized":"2023-10-06T01:41:13.351933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:37:14.679660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-10-06T01:41:12.348510+0000","last_change":"2023-10-06T01:41:12.341922+0000","last_active":"2023-10-06T01:41:12.348510+0000","last_peered":"2023-10-06T01:41:12.348510+0000","last_clean":"2023-10-06T01:41:12.348510+0000","last_became_active":"2023-10-06T01:41:12.341543+0000","last_became_peered":"2023-10-06T01:41:12.341543+0000","last_unstale":"2023-10-06T01:41:12.348510+0000","last_undegraded":"2023-10-06T01:41:12.348510+0000","last_fullsized":"2023-10-06T01:41:12.348510+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:13:14.004577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.345274+0000","last_change":"2023-10-06T01:41:13.345274+0000","last_active":"2023-10-06T01:41:13.345274+0000","last_peered":"2023-10-06T01:41:13.345274+0000","last_clean":"2023-10-06T01:41:13.345274+0000","last_became_active":"2023-10-06T01:41:13.344992+0000","last_became_peered":"2023-10-06T01:41:13.344992+0000","last_unstale":"2023-10-06T01:41:13.345274+0000","last_undegraded":"2023-10-06T01:41:13.345274+0000","last_fullsized":"2023-10-06T01:41:13.345274+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:39:03.091509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.339176+0000","last_change":"2023-10-06T01:41:12.343531+0000","last_active":"2023-10-06T01:41:13.339176+0000","last_peered":"2023-10-06T01:41:13.339176+0000","last_clean":"2023-10-06T01:41:13.339176+0000","last_became_active":"2023-10-06T01:41:12.343256+0000","last_became_peered":"2023-10-06T01:41:12.343256+0000","last_unstale":"2023-10-06T01:41:13.339176+0000","last_undegraded":"2023-10-06T01:41:13.339176+0000","last_fullsized":"2023-10-06T01:41:13.339176+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:16:14.826869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.357361+0000","last_change":"2023-10-06T01:41:13.357361+0000","last_active":"2023-10-06T01:41:13.357361+0000","last_peered":"2023-10-06T01:41:13.357361+0000","last_clean":"2023-10-06T01:41:13.357361+0000","last_became_active":"2023-10-06T01:41:13.356980+0000","last_became_peered":"2023-10-06T01:41:13.356980+0000","last_unstale":"2023-10-06T01:41:13.357361+0000","last_undegraded":"2023-10-06T01:41:13.357361+0000","last_fullsized":"2023-10-06T01:41:13.357361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:42:51.581939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"19'3","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.355632+0000","last_change":"2023-10-06T01:41:12.350010+0000","last_active":"2023-10-06T01:41:13.355632+0000","last_peered":"2023-10-06T01:41:13.355632+0000","last_clean":"2023-10-06T01:41:13.355632+0000","last_became_active":"2023-10-06T01:41:12.349758+0000","last_became_peered":"2023-10-06T01:41:12.349758+0000","last_unstale":"2023-10-06T01:41:13.355632+0000","last_undegraded":"2023-10-06T01:41:13.355632+0000","last_fullsized":"2023-10-06T01:41:13.355632+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:12:51.734395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352993+0000","last_change":"2023-10-06T01:41:13.348953+0000","last_active":"2023-10-06T01:41:13.352993+0000","last_peered":"2023-10-06T01:41:13.352993+0000","last_clean":"2023-10-06T01:41:13.352993+0000","last_became_active":"2023-10-06T01:41:13.348683+0000","last_became_peered":"2023-10-06T01:41:13.348683+0000","last_unstale":"2023-10-06T01:41:13.352993+0000","last_undegraded":"2023-10-06T01:41:13.352993+0000","last_fullsized":"2023-10-06T01:41:13.352993+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:42:46.214805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"19'6","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.357074+0000","last_change":"2023-10-06T01:41:12.340922+0000","last_active":"2023-10-06T01:41:13.357074+0000","last_peered":"2023-10-06T01:41:13.357074+0000","last_clean":"2023-10-06T01:41:13.357074+0000","last_became_active":"2023-10-06T01:41:12.340598+0000","last_became_peered":"2023-10-06T01:41:12.340598+0000","last_unstale":"2023-10-06T01:41:13.357074+0000","last_undegraded":"2023-10-06T01:41:13.357074+0000","last_fullsized":"2023-10-06T01:41:13.357074+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:06:19.029137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351653+0000","last_change":"2023-10-06T01:41:13.343918+0000","last_active":"2023-10-06T01:41:13.351653+0000","last_peered":"2023-10-06T01:41:13.351653+0000","last_clean":"2023-10-06T01:41:13.351653+0000","last_became_active":"2023-10-06T01:41:13.343621+0000","last_became_peered":"2023-10-06T01:41:13.343621+0000","last_unstale":"2023-10-06T01:41:13.351653+0000","last_undegraded":"2023-10-06T01:41:13.351653+0000","last_fullsized":"2023-10-06T01:41:13.351653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:42:44.799267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.350439+0000","last_change":"2023-10-06T01:41:12.343541+0000","last_active":"2023-10-06T01:41:13.350439+0000","last_peered":"2023-10-06T01:41:13.350439+0000","last_clean":"2023-10-06T01:41:13.350439+0000","last_became_active":"2023-10-06T01:41:12.342645+0000","last_became_peered":"2023-10-06T01:41:12.342645+0000","last_unstale":"2023-10-06T01:41:13.350439+0000","last_undegraded":"2023-10-06T01:41:13.350439+0000","last_fullsized":"2023-10-06T01:41:13.350439+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:16:14.826869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352574+0000","last_change":"2023-10-06T01:41:13.352574+0000","last_active":"2023-10-06T01:41:13.352574+0000","last_peered":"2023-10-06T01:41:13.352574+0000","last_clean":"2023-10-06T01:41:13.352574+0000","last_became_active":"2023-10-06T01:41:13.352300+0000","last_became_peered":"2023-10-06T01:41:13.352300+0000","last_unstale":"2023-10-06T01:41:13.352574+0000","last_undegraded":"2023-10-06T01:41:13.352574+0000","last_fullsized":"2023-10-06T01:41:13.352574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:27:54.074728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-10-06T01:41:12.351884+0000","last_change":"2023-10-06T01:41:12.344679+0000","last_active":"2023-10-06T01:41:12.351884+0000","last_peered":"2023-10-06T01:41:12.351884+0000","last_clean":"2023-10-06T01:41:12.351884+0000","last_became_active":"2023-10-06T01:41:12.344430+0000","last_became_peered":"2023-10-06T01:41:12.344430+0000","last_unstale":"2023-10-06T01:41:12.351884+0000","last_undegraded":"2023-10-06T01:41:12.351884+0000","last_fullsized":"2023-10-06T01:41:12.351884+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:27:01.582307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.360786+0000","last_change":"2023-10-06T01:41:13.357148+0000","last_active":"2023-10-06T01:41:13.360786+0000","last_peered":"2023-10-06T01:41:13.360786+0000","last_clean":"2023-10-06T01:41:13.360786+0000","last_became_active":"2023-10-06T01:41:13.356843+0000","last_became_peered":"2023-10-06T01:41:13.356843+0000","last_unstale":"2023-10-06T01:41:13.360786+0000","last_undegraded":"2023-10-06T01:41:13.360786+0000","last_fullsized":"2023-10-06T01:41:13.360786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T01:51:57.229301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.334167+0000","last_change":"2023-10-06T01:41:12.344068+0000","last_active":"2023-10-06T01:41:13.334167+0000","last_peered":"2023-10-06T01:41:13.334167+0000","last_clean":"2023-10-06T01:41:13.334167+0000","last_became_active":"2023-10-06T01:41:12.342562+0000","last_became_peered":"2023-10-06T01:41:12.342562+0000","last_unstale":"2023-10-06T01:41:13.334167+0000","last_undegraded":"2023-10-06T01:41:13.334167+0000","last_fullsized":"2023-10-06T01:41:13.334167+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:37:13.672716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.343508+0000","last_change":"2023-10-06T01:41:13.343508+0000","last_active":"2023-10-06T01:41:13.343508+0000","last_peered":"2023-10-06T01:41:13.343508+0000","last_clean":"2023-10-06T01:41:13.343508+0000","last_became_active":"2023-10-06T01:41:13.343249+0000","last_became_peered":"2023-10-06T01:41:13.343249+0000","last_unstale":"2023-10-06T01:41:13.343508+0000","last_undegraded":"2023-10-06T01:41:13.343508+0000","last_fullsized":"2023-10-06T01:41:13.343508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:03:37.385158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"19'8","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.347028+0000","last_change":"2023-10-06T01:41:12.342949+0000","last_active":"2023-10-06T01:41:13.347028+0000","last_peered":"2023-10-06T01:41:13.347028+0000","last_clean":"2023-10-06T01:41:13.347028+0000","last_became_active":"2023-10-06T01:41:12.342687+0000","last_became_peered":"2023-10-06T01:41:12.342687+0000","last_unstale":"2023-10-06T01:41:13.347028+0000","last_undegraded":"2023-10-06T01:41:13.347028+0000","last_fullsized":"2023-10-06T01:41:13.347028+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:37:52.143727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354861+0000","last_change":"2023-10-06T01:41:13.354861+0000","last_active":"2023-10-06T01:41:13.354861+0000","last_peered":"2023-10-06T01:41:13.354861+0000","last_clean":"2023-10-06T01:41:13.354861+0000","last_became_active":"2023-10-06T01:41:13.354190+0000","last_became_peered":"2023-10-06T01:41:13.354190+0000","last_unstale":"2023-10-06T01:41:13.354861+0000","last_undegraded":"2023-10-06T01:41:13.354861+0000","last_fullsized":"2023-10-06T01:41:13.354861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:45:07.571069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"19'1","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.349986+0000","last_change":"2023-10-06T01:41:12.344048+0000","last_active":"2023-10-06T01:41:13.349986+0000","last_peered":"2023-10-06T01:41:13.349986+0000","last_clean":"2023-10-06T01:41:13.349986+0000","last_became_active":"2023-10-06T01:41:12.343532+0000","last_became_peered":"2023-10-06T01:41:12.343532+0000","last_unstale":"2023-10-06T01:41:13.349986+0000","last_undegraded":"2023-10-06T01:41:13.349986+0000","last_fullsized":"2023-10-06T01:41:13.349986+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:50:44.552768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354457+0000","last_change":"2023-10-06T01:41:13.351276+0000","last_active":"2023-10-06T01:41:13.354457+0000","last_peered":"2023-10-06T01:41:13.354457+0000","last_clean":"2023-10-06T01:41:13.354457+0000","last_became_active":"2023-10-06T01:41:13.351005+0000","last_became_peered":"2023-10-06T01:41:13.351005+0000","last_unstale":"2023-10-06T01:41:13.354457+0000","last_undegraded":"2023-10-06T01:41:13.354457+0000","last_fullsized":"2023-10-06T01:41:13.354457+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:42:51.581939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356755+0000","last_change":"2023-10-06T01:41:12.344159+0000","last_active":"2023-10-06T01:41:13.356755+0000","last_peered":"2023-10-06T01:41:13.356755+0000","last_clean":"2023-10-06T01:41:13.356755+0000","last_became_active":"2023-10-06T01:41:12.343593+0000","last_became_peered":"2023-10-06T01:41:12.343593+0000","last_unstale":"2023-10-06T01:41:13.356755+0000","last_undegraded":"2023-10-06T01:41:13.356755+0000","last_fullsized":"2023-10-06T01:41:13.356755+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:27:01.582307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354635+0000","last_change":"2023-10-06T01:41:13.349120+0000","last_active":"2023-10-06T01:41:13.354635+0000","last_peered":"2023-10-06T01:41:13.354635+0000","last_clean":"2023-10-06T01:41:13.354635+0000","last_became_active":"2023-10-06T01:41:13.348850+0000","last_became_peered":"2023-10-06T01:41:13.348850+0000","last_unstale":"2023-10-06T01:41:13.354635+0000","last_undegraded":"2023-10-06T01:41:13.354635+0000","last_fullsized":"2023-10-06T01:41:13.354635+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:27:32.273170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"19'1","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.358476+0000","last_change":"2023-10-06T01:41:12.344880+0000","last_active":"2023-10-06T01:41:13.358476+0000","last_peered":"2023-10-06T01:41:13.358476+0000","last_clean":"2023-10-06T01:41:13.358476+0000","last_became_active":"2023-10-06T01:41:12.344676+0000","last_became_peered":"2023-10-06T01:41:12.344676+0000","last_unstale":"2023-10-06T01:41:13.358476+0000","last_undegraded":"2023-10-06T01:41:13.358476+0000","last_fullsized":"2023-10-06T01:41:13.358476+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:21:09.833061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.353951+0000","last_change":"2023-10-06T01:41:13.348117+0000","last_active":"2023-10-06T01:41:13.353951+0000","last_peered":"2023-10-06T01:41:13.353951+0000","last_clean":"2023-10-06T01:41:13.353951+0000","last_became_active":"2023-10-06T01:41:13.347872+0000","last_became_peered":"2023-10-06T01:41:13.347872+0000","last_unstale":"2023-10-06T01:41:13.353951+0000","last_undegraded":"2023-10-06T01:41:13.353951+0000","last_fullsized":"2023-10-06T01:41:13.353951+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:21:10.840006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.338677+0000","last_change":"2023-10-06T01:41:12.341362+0000","last_active":"2023-10-06T01:41:13.338677+0000","last_peered":"2023-10-06T01:41:13.338677+0000","last_clean":"2023-10-06T01:41:13.338677+0000","last_became_active":"2023-10-06T01:41:12.340855+0000","last_became_peered":"2023-10-06T01:41:12.340855+0000","last_unstale":"2023-10-06T01:41:13.338677+0000","last_undegraded":"2023-10-06T01:41:13.338677+0000","last_fullsized":"2023-10-06T01:41:13.338677+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:15:40.841990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.344966+0000","last_change":"2023-10-06T01:41:13.344966+0000","last_active":"2023-10-06T01:41:13.344966+0000","last_peered":"2023-10-06T01:41:13.344966+0000","last_clean":"2023-10-06T01:41:13.344966+0000","last_became_active":"2023-10-06T01:41:13.344700+0000","last_became_peered":"2023-10-06T01:41:13.344700+0000","last_unstale":"2023-10-06T01:41:13.344966+0000","last_undegraded":"2023-10-06T01:41:13.344966+0000","last_fullsized":"2023-10-06T01:41:13.344966+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:42:23.199959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"19'2","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.346245+0000","last_change":"2023-10-06T01:41:12.343169+0000","last_active":"2023-10-06T01:41:13.346245+0000","last_peered":"2023-10-06T01:41:13.346245+0000","last_clean":"2023-10-06T01:41:13.346245+0000","last_became_active":"2023-10-06T01:41:12.342474+0000","last_became_peered":"2023-10-06T01:41:12.342474+0000","last_unstale":"2023-10-06T01:41:13.346245+0000","last_undegraded":"2023-10-06T01:41:13.346245+0000","last_fullsized":"2023-10-06T01:41:13.346245+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:27:01.582307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351608+0000","last_change":"2023-10-06T01:41:13.343671+0000","last_active":"2023-10-06T01:41:13.351608+0000","last_peered":"2023-10-06T01:41:13.351608+0000","last_clean":"2023-10-06T01:41:13.351608+0000","last_became_active":"2023-10-06T01:41:13.341868+0000","last_became_peered":"2023-10-06T01:41:13.341868+0000","last_unstale":"2023-10-06T01:41:13.351608+0000","last_undegraded":"2023-10-06T01:41:13.351608+0000","last_fullsized":"2023-10-06T01:41:13.351608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T01:51:57.229301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.329621+0000","last_change":"2023-10-06T01:41:12.339878+0000","last_active":"2023-10-06T01:41:13.329621+0000","last_peered":"2023-10-06T01:41:13.329621+0000","last_clean":"2023-10-06T01:41:13.329621+0000","last_became_active":"2023-10-06T01:41:12.338935+0000","last_became_peered":"2023-10-06T01:41:12.338935+0000","last_unstale":"2023-10-06T01:41:13.329621+0000","last_undegraded":"2023-10-06T01:41:13.329621+0000","last_fullsized":"2023-10-06T01:41:13.329621+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:44:53.014481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351640+0000","last_change":"2023-10-06T01:41:13.344292+0000","last_active":"2023-10-06T01:41:13.351640+0000","last_peered":"2023-10-06T01:41:13.351640+0000","last_clean":"2023-10-06T01:41:13.351640+0000","last_became_active":"2023-10-06T01:41:13.343947+0000","last_became_peered":"2023-10-06T01:41:13.343947+0000","last_unstale":"2023-10-06T01:41:13.351640+0000","last_undegraded":"2023-10-06T01:41:13.351640+0000","last_fullsized":"2023-10-06T01:41:13.351640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:45:07.571069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.331713+0000","last_change":"2023-10-06T01:41:12.336355+0000","last_active":"2023-10-06T01:41:13.331713+0000","last_peered":"2023-10-06T01:41:13.331713+0000","last_clean":"2023-10-06T01:41:13.331713+0000","last_became_active":"2023-10-06T01:41:12.336064+0000","last_became_peered":"2023-10-06T01:41:12.336064+0000","last_unstale":"2023-10-06T01:41:13.331713+0000","last_undegraded":"2023-10-06T01:41:13.331713+0000","last_fullsized":"2023-10-06T01:41:13.331713+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:16:14.826869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354641+0000","last_change":"2023-10-06T01:41:13.346404+0000","last_active":"2023-10-06T01:41:13.354641+0000","last_peered":"2023-10-06T01:41:13.354641+0000","last_clean":"2023-10-06T01:41:13.354641+0000","last_became_active":"2023-10-06T01:41:13.343805+0000","last_became_peered":"2023-10-06T01:41:13.343805+0000","last_unstale":"2023-10-06T01:41:13.354641+0000","last_undegraded":"2023-10-06T01:41:13.354641+0000","last_fullsized":"2023-10-06T01:41:13.354641+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:11:26.794025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-10-06T01:41:12.349596+0000","last_change":"2023-10-06T01:41:12.345661+0000","last_active":"2023-10-06T01:41:12.349596+0000","last_peered":"2023-10-06T01:41:12.349596+0000","last_clean":"2023-10-06T01:41:12.349596+0000","last_became_active":"2023-10-06T01:41:12.345363+0000","last_became_peered":"2023-10-06T01:41:12.345363+0000","last_unstale":"2023-10-06T01:41:12.349596+0000","last_undegraded":"2023-10-06T01:41:12.349596+0000","last_fullsized":"2023-10-06T01:41:12.349596+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:50:44.552768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356650+0000","last_change":"2023-10-06T01:41:13.353633+0000","last_active":"2023-10-06T01:41:13.356650+0000","last_peered":"2023-10-06T01:41:13.356650+0000","last_clean":"2023-10-06T01:41:13.356650+0000","last_became_active":"2023-10-06T01:41:13.353318+0000","last_became_peered":"2023-10-06T01:41:13.353318+0000","last_unstale":"2023-10-06T01:41:13.356650+0000","last_undegraded":"2023-10-06T01:41:13.356650+0000","last_fullsized":"2023-10-06T01:41:13.356650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:42:23.199959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"19'2","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354459+0000","last_change":"2023-10-06T01:41:12.340463+0000","last_active":"2023-10-06T01:41:13.354459+0000","last_peered":"2023-10-06T01:41:13.354459+0000","last_clean":"2023-10-06T01:41:13.354459+0000","last_became_active":"2023-10-06T01:41:12.340186+0000","last_became_peered":"2023-10-06T01:41:12.340186+0000","last_unstale":"2023-10-06T01:41:13.354459+0000","last_undegraded":"2023-10-06T01:41:13.354459+0000","last_fullsized":"2023-10-06T01:41:13.354459+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:16:14.826869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.330869+0000","last_change":"2023-10-06T01:41:07.454295+0000","last_active":"2023-10-06T01:41:13.330869+0000","last_peered":"2023-10-06T01:41:13.330869+0000","last_clean":"2023-10-06T01:41:13.330869+0000","last_became_active":"2023-10-06T01:41:07.454034+0000","last_became_peered":"2023-10-06T01:41:07.454034+0000","last_unstale":"2023-10-06T01:41:13.330869+0000","last_undegraded":"2023-10-06T01:41:13.330869+0000","last_fullsized":"2023-10-06T01:41:13.330869+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:32:58.914183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351289+0000","last_change":"2023-10-06T01:41:13.351289+0000","last_active":"2023-10-06T01:41:13.351289+0000","last_peered":"2023-10-06T01:41:13.351289+0000","last_clean":"2023-10-06T01:41:13.351289+0000","last_became_active":"2023-10-06T01:41:13.351022+0000","last_became_peered":"2023-10-06T01:41:13.351022+0000","last_unstale":"2023-10-06T01:41:13.351289+0000","last_undegraded":"2023-10-06T01:41:13.351289+0000","last_fullsized":"2023-10-06T01:41:13.351289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T01:51:57.229301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-10-06T01:41:12.327374+0000","last_change":"2023-10-06T01:41:07.261955+0000","last_active":"2023-10-06T01:41:12.327374+0000","last_peered":"2023-10-06T01:41:12.327374+0000","last_clean":"2023-10-06T01:41:12.327374+0000","last_became_active":"2023-10-06T01:41:07.261692+0000","last_became_peered":"2023-10-06T01:41:07.261692+0000","last_unstale":"2023-10-06T01:41:12.327374+0000","last_undegraded":"2023-10-06T01:41:12.327374+0000","last_fullsized":"2023-10-06T01:41:12.327374+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-10-06T01:41:12.344691+0000","last_change":"2023-10-06T01:41:12.341312+0000","last_active":"2023-10-06T01:41:12.344691+0000","last_peered":"2023-10-06T01:41:12.344691+0000","last_clean":"2023-10-06T01:41:12.344691+0000","last_became_active":"2023-10-06T01:41:12.340997+0000","last_became_peered":"2023-10-06T01:41:12.340997+0000","last_unstale":"2023-10-06T01:41:12.344691+0000","last_undegraded":"2023-10-06T01:41:12.344691+0000","last_fullsized":"2023-10-06T01:41:12.344691+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:06:19.029137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351571+0000","last_change":"2023-10-06T01:41:13.344156+0000","last_active":"2023-10-06T01:41:13.351571+0000","last_peered":"2023-10-06T01:41:13.351571+0000","last_clean":"2023-10-06T01:41:13.351571+0000","last_became_active":"2023-10-06T01:41:13.343601+0000","last_became_peered":"2023-10-06T01:41:13.343601+0000","last_unstale":"2023-10-06T01:41:13.351571+0000","last_undegraded":"2023-10-06T01:41:13.351571+0000","last_fullsized":"2023-10-06T01:41:13.351571+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:21:42.851681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.336375+0000","last_change":"2023-10-06T01:41:07.258362+0000","last_active":"2023-10-06T01:41:13.336375+0000","last_peered":"2023-10-06T01:41:13.336375+0000","last_clean":"2023-10-06T01:41:13.336375+0000","last_became_active":"2023-10-06T01:41:07.258055+0000","last_became_peered":"2023-10-06T01:41:07.258055+0000","last_unstale":"2023-10-06T01:41:13.336375+0000","last_undegraded":"2023-10-06T01:41:13.336375+0000","last_fullsized":"2023-10-06T01:41:13.336375+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"16'117","reported_seq":163,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.334859+0000","last_change":"2023-10-06T01:41:07.261164+0000","last_active":"2023-10-06T01:41:13.334859+0000","last_peered":"2023-10-06T01:41:13.334859+0000","last_clean":"2023-10-06T01:41:13.334859+0000","last_became_active":"2023-10-06T01:41:07.260896+0000","last_became_peered":"2023-10-06T01:41:07.260896+0000","last_unstale":"2023-10-06T01:41:13.334859+0000","last_undegraded":"2023-10-06T01:41:13.334859+0000","last_fullsized":"2023-10-06T01:41:13.334859+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:05:22.934401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-10-06T01:41:12.348536+0000","last_change":"2023-10-06T01:41:12.343057+0000","last_active":"2023-10-06T01:41:12.348536+0000","last_peered":"2023-10-06T01:41:12.348536+0000","last_clean":"2023-10-06T01:41:12.348536+0000","last_became_active":"2023-10-06T01:41:12.342793+0000","last_became_peered":"2023-10-06T01:41:12.342793+0000","last_unstale":"2023-10-06T01:41:12.348536+0000","last_undegraded":"2023-10-06T01:41:12.348536+0000","last_fullsized":"2023-10-06T01:41:12.348536+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:15:40.841990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.346571+0000","last_change":"2023-10-06T01:41:13.346571+0000","last_active":"2023-10-06T01:41:13.346571+0000","last_peered":"2023-10-06T01:41:13.346571+0000","last_clean":"2023-10-06T01:41:13.346571+0000","last_became_active":"2023-10-06T01:41:13.346077+0000","last_became_peered":"2023-10-06T01:41:13.346077+0000","last_unstale":"2023-10-06T01:41:13.346571+0000","last_undegraded":"2023-10-06T01:41:13.346571+0000","last_fullsized":"2023-10-06T01:41:13.346571+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:42:51.581939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.333951+0000","last_change":"2023-10-06T01:41:07.456511+0000","last_active":"2023-10-06T01:41:13.333951+0000","last_peered":"2023-10-06T01:41:13.333951+0000","last_clean":"2023-10-06T01:41:13.333951+0000","last_became_active":"2023-10-06T01:41:07.456024+0000","last_became_peered":"2023-10-06T01:41:07.456024+0000","last_unstale":"2023-10-06T01:41:13.333951+0000","last_undegraded":"2023-10-06T01:41:13.333951+0000","last_fullsized":"2023-10-06T01:41:13.333951+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:51:02.949231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.336635+0000","last_change":"2023-10-06T01:41:12.343382+0000","last_active":"2023-10-06T01:41:13.336635+0000","last_peered":"2023-10-06T01:41:13.336635+0000","last_clean":"2023-10-06T01:41:13.336635+0000","last_became_active":"2023-10-06T01:41:12.343088+0000","last_became_peered":"2023-10-06T01:41:12.343088+0000","last_unstale":"2023-10-06T01:41:13.336635+0000","last_undegraded":"2023-10-06T01:41:13.336635+0000","last_fullsized":"2023-10-06T01:41:13.336635+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:44:53.014481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.350157+0000","last_change":"2023-10-06T01:41:13.346852+0000","last_active":"2023-10-06T01:41:13.350157+0000","last_peered":"2023-10-06T01:41:13.350157+0000","last_clean":"2023-10-06T01:41:13.350157+0000","last_became_active":"2023-10-06T01:41:13.346571+0000","last_became_peered":"2023-10-06T01:41:13.346571+0000","last_unstale":"2023-10-06T01:41:13.350157+0000","last_undegraded":"2023-10-06T01:41:13.350157+0000","last_fullsized":"2023-10-06T01:41:13.350157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:25:04.563235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.334453+0000","last_change":"2023-10-06T01:41:07.456273+0000","last_active":"2023-10-06T01:41:13.334453+0000","last_peered":"2023-10-06T01:41:13.334453+0000","last_clean":"2023-10-06T01:41:13.334453+0000","last_became_active":"2023-10-06T01:41:07.456000+0000","last_became_peered":"2023-10-06T01:41:07.456000+0000","last_unstale":"2023-10-06T01:41:13.334453+0000","last_undegraded":"2023-10-06T01:41:13.334453+0000","last_fullsized":"2023-10-06T01:41:13.334453+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:01:42.633388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-10-06T01:41:12.349428+0000","last_change":"2023-10-06T01:41:12.342215+0000","last_active":"2023-10-06T01:41:12.349428+0000","last_peered":"2023-10-06T01:41:12.349428+0000","last_clean":"2023-10-06T01:41:12.349428+0000","last_became_active":"2023-10-06T01:41:12.341493+0000","last_became_peered":"2023-10-06T01:41:12.341493+0000","last_unstale":"2023-10-06T01:41:12.349428+0000","last_undegraded":"2023-10-06T01:41:12.349428+0000","last_fullsized":"2023-10-06T01:41:12.349428+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:49:54.658088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352134+0000","last_change":"2023-10-06T01:41:13.346680+0000","last_active":"2023-10-06T01:41:13.352134+0000","last_peered":"2023-10-06T01:41:13.352134+0000","last_clean":"2023-10-06T01:41:13.352134+0000","last_became_active":"2023-10-06T01:41:13.346326+0000","last_became_peered":"2023-10-06T01:41:13.346326+0000","last_unstale":"2023-10-06T01:41:13.352134+0000","last_undegraded":"2023-10-06T01:41:13.352134+0000","last_fullsized":"2023-10-06T01:41:13.352134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:12:52.741339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.338955+0000","last_change":"2023-10-06T01:41:07.456688+0000","last_active":"2023-10-06T01:41:13.338955+0000","last_peered":"2023-10-06T01:41:13.338955+0000","last_clean":"2023-10-06T01:41:13.338955+0000","last_became_active":"2023-10-06T01:41:07.456304+0000","last_became_peered":"2023-10-06T01:41:07.456304+0000","last_unstale":"2023-10-06T01:41:13.338955+0000","last_undegraded":"2023-10-06T01:41:13.338955+0000","last_fullsized":"2023-10-06T01:41:13.338955+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.332203+0000","last_change":"2023-10-06T01:41:12.342354+0000","last_active":"2023-10-06T01:41:13.332203+0000","last_peered":"2023-10-06T01:41:13.332203+0000","last_clean":"2023-10-06T01:41:13.332203+0000","last_became_active":"2023-10-06T01:41:12.342066+0000","last_became_peered":"2023-10-06T01:41:12.342066+0000","last_unstale":"2023-10-06T01:41:13.332203+0000","last_undegraded":"2023-10-06T01:41:13.332203+0000","last_fullsized":"2023-10-06T01:41:13.332203+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:15:40.841990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352905+0000","last_change":"2023-10-06T01:41:13.348241+0000","last_active":"2023-10-06T01:41:13.352905+0000","last_peered":"2023-10-06T01:41:13.352905+0000","last_clean":"2023-10-06T01:41:13.352905+0000","last_became_active":"2023-10-06T01:41:13.347979+0000","last_became_peered":"2023-10-06T01:41:13.347979+0000","last_unstale":"2023-10-06T01:41:13.352905+0000","last_undegraded":"2023-10-06T01:41:13.352905+0000","last_fullsized":"2023-10-06T01:41:13.352905+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:41:59.186469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.329416+0000","last_change":"2023-10-06T01:41:07.456388+0000","last_active":"2023-10-06T01:41:13.329416+0000","last_peered":"2023-10-06T01:41:13.329416+0000","last_clean":"2023-10-06T01:41:13.329416+0000","last_became_active":"2023-10-06T01:41:07.456045+0000","last_became_peered":"2023-10-06T01:41:07.456045+0000","last_unstale":"2023-10-06T01:41:13.329416+0000","last_undegraded":"2023-10-06T01:41:13.329416+0000","last_fullsized":"2023-10-06T01:41:13.329416+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.338387+0000","last_change":"2023-10-06T01:41:12.344268+0000","last_active":"2023-10-06T01:41:13.338387+0000","last_peered":"2023-10-06T01:41:13.338387+0000","last_clean":"2023-10-06T01:41:13.338387+0000","last_became_active":"2023-10-06T01:41:12.343996+0000","last_became_peered":"2023-10-06T01:41:12.343996+0000","last_unstale":"2023-10-06T01:41:13.338387+0000","last_undegraded":"2023-10-06T01:41:13.338387+0000","last_fullsized":"2023-10-06T01:41:13.338387+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:37:52.143727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.346184+0000","last_change":"2023-10-06T01:41:13.346184+0000","last_active":"2023-10-06T01:41:13.346184+0000","last_peered":"2023-10-06T01:41:13.346184+0000","last_clean":"2023-10-06T01:41:13.346184+0000","last_became_active":"2023-10-06T01:41:13.345570+0000","last_became_peered":"2023-10-06T01:41:13.345570+0000","last_unstale":"2023-10-06T01:41:13.346184+0000","last_undegraded":"2023-10-06T01:41:13.346184+0000","last_fullsized":"2023-10-06T01:41:13.346184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:42:46.214805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.330470+0000","last_change":"2023-10-06T01:41:07.258162+0000","last_active":"2023-10-06T01:41:13.330470+0000","last_peered":"2023-10-06T01:41:13.330470+0000","last_clean":"2023-10-06T01:41:13.330470+0000","last_became_active":"2023-10-06T01:41:07.257795+0000","last_became_peered":"2023-10-06T01:41:07.257795+0000","last_unstale":"2023-10-06T01:41:13.330470+0000","last_undegraded":"2023-10-06T01:41:13.330470+0000","last_fullsized":"2023-10-06T01:41:13.330470+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:05:22.934401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.337002+0000","last_change":"2023-10-06T01:41:12.341990+0000","last_active":"2023-10-06T01:41:13.337002+0000","last_peered":"2023-10-06T01:41:13.337002+0000","last_clean":"2023-10-06T01:41:13.337002+0000","last_became_active":"2023-10-06T01:41:12.341680+0000","last_became_peered":"2023-10-06T01:41:12.341680+0000","last_unstale":"2023-10-06T01:41:13.337002+0000","last_undegraded":"2023-10-06T01:41:13.337002+0000","last_fullsized":"2023-10-06T01:41:13.337002+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:50:44.552768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.349056+0000","last_change":"2023-10-06T01:41:13.344589+0000","last_active":"2023-10-06T01:41:13.349056+0000","last_peered":"2023-10-06T01:41:13.349056+0000","last_clean":"2023-10-06T01:41:13.349056+0000","last_became_active":"2023-10-06T01:41:13.344281+0000","last_became_peered":"2023-10-06T01:41:13.344281+0000","last_unstale":"2023-10-06T01:41:13.349056+0000","last_undegraded":"2023-10-06T01:41:13.349056+0000","last_fullsized":"2023-10-06T01:41:13.349056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:32:17.938309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"19'1","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.329842+0000","last_change":"2023-10-06T01:41:12.347655+0000","last_active":"2023-10-06T01:41:13.329842+0000","last_peered":"2023-10-06T01:41:13.329842+0000","last_clean":"2023-10-06T01:41:13.329842+0000","last_became_active":"2023-10-06T01:41:12.347353+0000","last_became_peered":"2023-10-06T01:41:12.347353+0000","last_unstale":"2023-10-06T01:41:13.329842+0000","last_undegraded":"2023-10-06T01:41:13.329842+0000","last_fullsized":"2023-10-06T01:41:13.329842+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:49:54.658088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352446+0000","last_change":"2023-10-06T01:41:13.349874+0000","last_active":"2023-10-06T01:41:13.352446+0000","last_peered":"2023-10-06T01:41:13.352446+0000","last_clean":"2023-10-06T01:41:13.352446+0000","last_became_active":"2023-10-06T01:41:13.349607+0000","last_became_peered":"2023-10-06T01:41:13.349607+0000","last_unstale":"2023-10-06T01:41:13.352446+0000","last_undegraded":"2023-10-06T01:41:13.352446+0000","last_fullsized":"2023-10-06T01:41:13.352446+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:06:20.036081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352575+0000","last_change":"2023-10-06T01:41:12.341511+0000","last_active":"2023-10-06T01:41:13.352575+0000","last_peered":"2023-10-06T01:41:13.352575+0000","last_clean":"2023-10-06T01:41:13.352575+0000","last_became_active":"2023-10-06T01:41:12.341216+0000","last_became_peered":"2023-10-06T01:41:12.341216+0000","last_unstale":"2023-10-06T01:41:13.352575+0000","last_undegraded":"2023-10-06T01:41:13.352575+0000","last_fullsized":"2023-10-06T01:41:13.352575+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:27:01.582307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351635+0000","last_change":"2023-10-06T01:41:13.343780+0000","last_active":"2023-10-06T01:41:13.351635+0000","last_peered":"2023-10-06T01:41:13.351635+0000","last_clean":"2023-10-06T01:41:13.351635+0000","last_became_active":"2023-10-06T01:41:13.343515+0000","last_became_peered":"2023-10-06T01:41:13.343515+0000","last_unstale":"2023-10-06T01:41:13.351635+0000","last_undegraded":"2023-10-06T01:41:13.351635+0000","last_fullsized":"2023-10-06T01:41:13.351635+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:38:58.407819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.353702+0000","last_change":"2023-10-06T01:41:12.341780+0000","last_active":"2023-10-06T01:41:13.353702+0000","last_peered":"2023-10-06T01:41:13.353702+0000","last_clean":"2023-10-06T01:41:13.353702+0000","last_became_active":"2023-10-06T01:41:12.340585+0000","last_became_peered":"2023-10-06T01:41:12.340585+0000","last_unstale":"2023-10-06T01:41:13.353702+0000","last_undegraded":"2023-10-06T01:41:13.353702+0000","last_fullsized":"2023-10-06T01:41:13.353702+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:49:54.658088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354558+0000","last_change":"2023-10-06T01:41:13.345543+0000","last_active":"2023-10-06T01:41:13.354558+0000","last_peered":"2023-10-06T01:41:13.354558+0000","last_clean":"2023-10-06T01:41:13.354558+0000","last_became_active":"2023-10-06T01:41:13.345286+0000","last_became_peered":"2023-10-06T01:41:13.345286+0000","last_unstale":"2023-10-06T01:41:13.354558+0000","last_undegraded":"2023-10-06T01:41:13.354558+0000","last_fullsized":"2023-10-06T01:41:13.354558+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:52:48.286318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"19'1","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.337057+0000","last_change":"2023-10-06T01:41:12.340776+0000","last_active":"2023-10-06T01:41:13.337057+0000","last_peered":"2023-10-06T01:41:13.337057+0000","last_clean":"2023-10-06T01:41:13.337057+0000","last_became_active":"2023-10-06T01:41:12.340294+0000","last_became_peered":"2023-10-06T01:41:12.340294+0000","last_unstale":"2023-10-06T01:41:13.337057+0000","last_undegraded":"2023-10-06T01:41:13.337057+0000","last_fullsized":"2023-10-06T01:41:13.337057+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:37:13.672716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.355273+0000","last_change":"2023-10-06T01:41:13.355273+0000","last_active":"2023-10-06T01:41:13.355273+0000","last_peered":"2023-10-06T01:41:13.355273+0000","last_clean":"2023-10-06T01:41:13.355273+0000","last_became_active":"2023-10-06T01:41:13.354977+0000","last_became_peered":"2023-10-06T01:41:13.354977+0000","last_unstale":"2023-10-06T01:41:13.355273+0000","last_undegraded":"2023-10-06T01:41:13.355273+0000","last_fullsized":"2023-10-06T01:41:13.355273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:21:42.851681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"19'4","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.345488+0000","last_change":"2023-10-06T01:41:12.338431+0000","last_active":"2023-10-06T01:41:13.345488+0000","last_peered":"2023-10-06T01:41:13.345488+0000","last_clean":"2023-10-06T01:41:13.345488+0000","last_became_active":"2023-10-06T01:41:12.338081+0000","last_became_peered":"2023-10-06T01:41:12.338081+0000","last_unstale":"2023-10-06T01:41:13.345488+0000","last_undegraded":"2023-10-06T01:41:13.345488+0000","last_fullsized":"2023-10-06T01:41:13.345488+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:16:14.826869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351157+0000","last_change":"2023-10-06T01:41:13.342212+0000","last_active":"2023-10-06T01:41:13.351157+0000","last_peered":"2023-10-06T01:41:13.351157+0000","last_clean":"2023-10-06T01:41:13.351157+0000","last_became_active":"2023-10-06T01:41:13.341948+0000","last_became_peered":"2023-10-06T01:41:13.341948+0000","last_unstale":"2023-10-06T01:41:13.351157+0000","last_undegraded":"2023-10-06T01:41:13.351157+0000","last_fullsized":"2023-10-06T01:41:13.351157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:18:23.688664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354188+0000","last_change":"2023-10-06T01:41:13.348482+0000","last_active":"2023-10-06T01:41:13.354188+0000","last_peered":"2023-10-06T01:41:13.354188+0000","last_clean":"2023-10-06T01:41:13.354188+0000","last_became_active":"2023-10-06T01:41:13.348075+0000","last_became_peered":"2023-10-06T01:41:13.348075+0000","last_unstale":"2023-10-06T01:41:13.354188+0000","last_undegraded":"2023-10-06T01:41:13.354188+0000","last_fullsized":"2023-10-06T01:41:13.354188+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:25:04.563235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.355216+0000","last_change":"2023-10-06T01:41:13.355216+0000","last_active":"2023-10-06T01:41:13.355216+0000","last_peered":"2023-10-06T01:41:13.355216+0000","last_clean":"2023-10-06T01:41:13.355216+0000","last_became_active":"2023-10-06T01:41:13.354853+0000","last_became_peered":"2023-10-06T01:41:13.354853+0000","last_unstale":"2023-10-06T01:41:13.355216+0000","last_undegraded":"2023-10-06T01:41:13.355216+0000","last_fullsized":"2023-10-06T01:41:13.355216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:42:44.799267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351163+0000","last_change":"2023-10-06T01:41:13.342888+0000","last_active":"2023-10-06T01:41:13.351163+0000","last_peered":"2023-10-06T01:41:13.351163+0000","last_clean":"2023-10-06T01:41:13.351163+0000","last_became_active":"2023-10-06T01:41:13.342626+0000","last_became_peered":"2023-10-06T01:41:13.342626+0000","last_unstale":"2023-10-06T01:41:13.351163+0000","last_undegraded":"2023-10-06T01:41:13.351163+0000","last_fullsized":"2023-10-06T01:41:13.351163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:43:29.491858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.353822+0000","last_change":"2023-10-06T01:41:13.353822+0000","last_active":"2023-10-06T01:41:13.353822+0000","last_peered":"2023-10-06T01:41:13.353822+0000","last_clean":"2023-10-06T01:41:13.353822+0000","last_became_active":"2023-10-06T01:41:13.353292+0000","last_became_peered":"2023-10-06T01:41:13.353292+0000","last_unstale":"2023-10-06T01:41:13.353822+0000","last_undegraded":"2023-10-06T01:41:13.353822+0000","last_fullsized":"2023-10-06T01:41:13.353822+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:52:48.286318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356636+0000","last_change":"2023-10-06T01:41:13.353662+0000","last_active":"2023-10-06T01:41:13.356636+0000","last_peered":"2023-10-06T01:41:13.356636+0000","last_clean":"2023-10-06T01:41:13.356636+0000","last_became_active":"2023-10-06T01:41:13.353269+0000","last_became_peered":"2023-10-06T01:41:13.353269+0000","last_unstale":"2023-10-06T01:41:13.356636+0000","last_undegraded":"2023-10-06T01:41:13.356636+0000","last_fullsized":"2023-10-06T01:41:13.356636+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:11:44.715827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298096,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073744,"statfs":{"total":96636764160,"available":96331513856,"internally_reserved":0,"allocated":827392,"data_stored":666304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297484,"kb_used_data":196,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074356,"statfs":{"total":96636764160,"available":96332140544,"internally_reserved":0,"allocated":200704,"data_stored":72831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298068,"kb_used_data":780,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073772,"statfs":{"total":96636764160,"available":96331542528,"internally_reserved":0,"allocated":798720,"data_stored":664552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297476,"kb_used_data":188,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074364,"statfs":{"total":96636764160,"available":96332148736,"internally_reserved":0,"allocated":192512,"data_stored":66744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72899999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297488,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074352,"statfs":{"total":96636764160,"available":96332136448,"internally_reserved":0,"allocated":204800,"data_stored":74184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297476,"kb_used_data":188,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074364,"statfs":{"total":96636764160,"available":96332148736,"internally_reserved":0,"allocated":192512,"data_stored":66280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297508,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074332,"statfs":{"total":96636764160,"available":96332115968,"internally_reserved":0,"allocated":225280,"data_stored":75776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297492,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074348,"statfs":{"total":96636764160,"available":96332132352,"internally_reserved":0,"allocated":208896,"data_stored":73455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-06T01:41:18.215 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-10-06T01:41:18.367 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.367+0000 7fec188de700 1 Processor -- start 2023-10-06T01:41:18.368 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.367+0000 7fec188de700 1 -- start start 2023-10-06T01:41:18.371 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.368+0000 7fec188de700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fec14142d90 0x7fec1406a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:18.371 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.368+0000 7fec188de700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec14069f50 con 0x7fec14143160 2023-10-06T01:41:18.371 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.368+0000 7fec188de700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec1406a920 con 0x7fec140cc130 2023-10-06T01:41:18.371 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.368+0000 7fec188de700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec14069340 con 0x7fec14142d90 2023-10-06T01:41:18.371 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.368+0000 7fec11d9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fec14143160 0x7fec140c43c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46206/0 (socket says 172.21.15.27:46206) 2023-10-06T01:41:18.371 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.368+0000 7fec11d9b700 1 -- 172.21.15.27:0/4038498963 learned_addr learned my addr 172.21.15.27:0/4038498963 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:18.372 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.368+0000 7fec1259c700 1 --2- 172.21.15.27:0/4038498963 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fec14142d90 0x7fec1406a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:18.372 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.369+0000 7fec1159a700 1 -- 172.21.15.27:0/4038498963 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4005171226 0 0) 0x7fec14069f50 con 0x7fec14143160 2023-10-06T01:41:18.372 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.369+0000 7fec1159a700 1 -- 172.21.15.27:0/4038498963 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febfc003570 con 0x7fec14143160 2023-10-06T01:41:18.372 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.369+0000 7fec1159a700 1 -- 172.21.15.27:0/4038498963 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 732255028 0 0) 0x7febfc003570 con 0x7fec14143160 2023-10-06T01:41:18.372 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.369+0000 7fec1159a700 1 -- 172.21.15.27:0/4038498963 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fec14142d90 msgr2=0x7fec1406a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:18.372 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.369+0000 7fec1159a700 1 --2- 172.21.15.27:0/4038498963 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fec14142d90 0x7fec1406a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:18.372 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.369+0000 7fec1159a700 1 -- 172.21.15.27:0/4038498963 >> v1:172.21.15.124:6789/0 conn(0x7fec140cc130 legacy=0x7fec140c8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:18.372 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.369+0000 7fec1159a700 1 -- 172.21.15.27:0/4038498963 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec14069620 con 0x7fec14143160 2023-10-06T01:41:18.372 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.369+0000 7fec1159a700 1 -- 172.21.15.27:0/4038498963 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2619078944 0 0) 0x7fec08001470 con 0x7fec14143160 2023-10-06T01:41:18.372 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.369+0000 7fec1159a700 1 -- 172.21.15.27:0/4038498963 >> v1:172.21.15.27:6789/0 conn(0x7fec14143160 legacy=0x7fec140c43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:18.372 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.369+0000 7fec1159a700 1 --2- 172.21.15.27:0/4038498963 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7febfc003a90 0x7febfc003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:18.373 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.369+0000 7fec1159a700 1 --2- 172.21.15.27:0/4038498963 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fec14142d90 0x7febfc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:18.373 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.369+0000 7fec1159a700 1 --2- 172.21.15.27:0/4038498963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febfc004d50 0x7febfc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:18.373 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.369+0000 7fec1159a700 1 -- 172.21.15.27:0/4038498963 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec14069340 con 0x7febfc004d50 2023-10-06T01:41:18.373 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.369+0000 7fec1159a700 1 -- 172.21.15.27:0/4038498963 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febfc00bb70 con 0x7febfc003a90 2023-10-06T01:41:18.373 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.369+0000 7fec1159a700 1 -- 172.21.15.27:0/4038498963 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febfc00bd30 con 0x7fec14142d90 2023-10-06T01:41:18.373 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.369+0000 7fec12d9d700 1 --2- 172.21.15.27:0/4038498963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febfc004d50 0x7febfc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:18.373 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.369+0000 7fec1259c700 1 --2- 172.21.15.27:0/4038498963 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7febfc003a90 0x7febfc003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:18.373 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.369+0000 7fec11d9b700 1 --2- 172.21.15.27:0/4038498963 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fec14142d90 0x7febfc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:18.373 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.369+0000 7fec12d9d700 1 -- 172.21.15.27:0/4038498963 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fec14142d90 msgr2=0x7febfc004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:18.373 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.369+0000 7fec12d9d700 1 --2- 172.21.15.27:0/4038498963 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fec14142d90 0x7febfc004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:18.373 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.369+0000 7fec12d9d700 1 -- 172.21.15.27:0/4038498963 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7febfc003a90 msgr2=0x7febfc003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:18.374 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.369+0000 7fec12d9d700 1 --2- 172.21.15.27:0/4038498963 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7febfc003a90 0x7febfc003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:18.374 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.369+0000 7fec12d9d700 1 -- 172.21.15.27:0/4038498963 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febfc003570 con 0x7febfc004d50 2023-10-06T01:41:18.374 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.370+0000 7fec12d9d700 1 --2- 172.21.15.27:0/4038498963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febfc004d50 0x7febfc0051c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fec0c001fc0 tx=0x7fec0c001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcc2a74b2be3d3bb server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:18.374 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.370+0000 7fec1159a700 1 -- 172.21.15.27:0/4038498963 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0c001950 con 0x7febfc004d50 2023-10-06T01:41:18.374 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.370+0000 7fec1159a700 1 -- 172.21.15.27:0/4038498963 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec0c00c8b0 con 0x7febfc004d50 2023-10-06T01:41:18.374 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.370+0000 7fec1159a700 1 -- 172.21.15.27:0/4038498963 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0c00cc50 con 0x7febfc004d50 2023-10-06T01:41:18.374 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.370+0000 7fec188de700 1 -- 172.21.15.27:0/4038498963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febfc004d50 msgr2=0x7febfc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:18.374 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.370+0000 7fec188de700 1 --2- 172.21.15.27:0/4038498963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7febfc004d50 0x7febfc0051c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fec0c001fc0 tx=0x7fec0c001640 comp rx=0 tx=0).stop 2023-10-06T01:41:18.374 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.370+0000 7fec12d9d700 1 -- 172.21.15.27:0/4038498963 reap_dead start 2023-10-06T01:41:18.374 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.370+0000 7fec188de700 1 -- 172.21.15.27:0/4038498963 shutdown_connections 2023-10-06T01:41:18.374 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.370+0000 7fec188de700 1 -- 172.21.15.27:0/4038498963 >> 172.21.15.27:0/4038498963 conn(0x7fec140ce4d0 msgr2=0x7fec140ce8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:18.374 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.370+0000 7fec188de700 1 -- 172.21.15.27:0/4038498963 shutdown_connections 2023-10-06T01:41:18.375 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.370+0000 7fec188de700 1 -- 172.21.15.27:0/4038498963 wait complete. 2023-10-06T01:41:18.377 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.371+0000 7fec188de700 1 Processor -- start 2023-10-06T01:41:18.377 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.371+0000 7fec188de700 1 -- start start 2023-10-06T01:41:18.377 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.372+0000 7fec188de700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7febfc004d50 0x7fec14141250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:18.377 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.372+0000 7fec188de700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fec140c6fa0 0x7fec14169990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:18.377 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.372+0000 7fec188de700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec14169ed0 0x7fec1406c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:18.377 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.372+0000 7fec188de700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec140c6550 con 0x7fec14169ed0 2023-10-06T01:41:18.377 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.372+0000 7fec188de700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec140c6b10 con 0x7febfc004d50 2023-10-06T01:41:18.378 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.372+0000 7fec188de700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec1416a4c0 con 0x7fec140c6fa0 2023-10-06T01:41:18.378 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.372+0000 7fec12d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec14169ed0 0x7fec1406c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:18.378 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.372+0000 7fec12d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec14169ed0 0x7fec1406c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55806/0 (socket says 172.21.15.27:55806) 2023-10-06T01:41:18.378 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.372+0000 7fec12d9d700 1 -- 172.21.15.27:0/3935661762 learned_addr learned my addr 172.21.15.27:0/3935661762 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:18.378 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.372+0000 7fec11d9b700 1 --2- 172.21.15.27:0/3935661762 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fec140c6fa0 0x7fec14169990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:18.378 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.372+0000 7fec1259c700 1 --2- 172.21.15.27:0/3935661762 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7febfc004d50 0x7fec14141250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:18.378 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.373+0000 7fec12d9d700 1 -- 172.21.15.27:0/3935661762 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fec140c6fa0 msgr2=0x7fec14169990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:18.378 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.373+0000 7fec12d9d700 1 --2- 172.21.15.27:0/3935661762 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fec140c6fa0 0x7fec14169990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:18.378 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.373+0000 7fec12d9d700 1 -- 172.21.15.27:0/3935661762 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7febfc004d50 msgr2=0x7fec14141250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:18.378 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.373+0000 7fec12d9d700 1 --2- 172.21.15.27:0/3935661762 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7febfc004d50 0x7fec14141250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:18.379 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.373+0000 7fec12d9d700 1 -- 172.21.15.27:0/3935661762 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec1416a7a0 con 0x7fec14169ed0 2023-10-06T01:41:18.379 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.373+0000 7fec12d9d700 1 --2- 172.21.15.27:0/3935661762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec14169ed0 0x7fec1406c310 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fec0c00c6a0 tx=0x7fec0c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:18.379 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.373+0000 7fec027fc700 1 -- 172.21.15.27:0/3935661762 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0c001950 con 0x7fec14169ed0 2023-10-06T01:41:18.379 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.373+0000 7fec027fc700 1 -- 172.21.15.27:0/3935661762 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec0c00c6d0 con 0x7fec14169ed0 2023-10-06T01:41:18.379 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.373+0000 7fec027fc700 1 -- 172.21.15.27:0/3935661762 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0c01f640 con 0x7fec14169ed0 2023-10-06T01:41:18.379 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.373+0000 7fec188de700 1 -- 172.21.15.27:0/3935661762 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec140c67a0 con 0x7fec14169ed0 2023-10-06T01:41:18.379 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.373+0000 7fec188de700 1 -- 172.21.15.27:0/3935661762 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec140c8260 con 0x7fec14169ed0 2023-10-06T01:41:18.379 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.374+0000 7fec188de700 1 -- 172.21.15.27:0/3935661762 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec140c6ca0 con 0x7fec14169ed0 2023-10-06T01:41:18.379 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.374+0000 7fec027fc700 1 -- 172.21.15.27:0/3935661762 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fec0c01fa70 con 0x7fec14169ed0 2023-10-06T01:41:18.379 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.375+0000 7fec027fc700 1 --2- 172.21.15.27:0/3935661762 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7febf4066f60 0x7febf4069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:18.379 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.375+0000 7fec027fc700 1 -- 172.21.15.27:0/3935661762 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fec0c086380 con 0x7fec14169ed0 2023-10-06T01:41:18.379 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.375+0000 7fec1259c700 1 --2- 172.21.15.27:0/3935661762 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7febf4066f60 0x7febf4069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:18.380 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.375+0000 7fec1259c700 1 --2- 172.21.15.27:0/3935661762 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7febf4066f60 0x7febf4069410 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fec04006f00 tx=0x7fec040091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:18.380 INFO:tasks.ceph.mds.d.smithi124.stderr:2023-10-06T01:41:18.376+0000 7f43112ad700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-06T01:41:18.380 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.379+0000 7fec027fc700 1 -- 172.21.15.27:0/3935661762 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fec0c053490 con 0x7fec14169ed0 2023-10-06T01:41:18.518 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T01:41:18.520 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T01:41:18.654 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.654+0000 7fec188de700 1 -- 172.21.15.27:0/3935661762 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fec140c6550 con 0x7fec14169ed0 2023-10-06T01:41:18.655 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.654+0000 7fec027fc700 1 -- 172.21.15.27:0/3935661762 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1015 (secure 0 0 0) 0x7fec0c023410 con 0x7fec14169ed0 2023-10-06T01:41:18.655 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:18.655 INFO:teuthology.orchestra.run.smithi027.stdout:{"fsid":"20a7bff0-74f8-4f2f-b6c6-899e804bc965","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":24,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":20,"num_osds":8,"num_up_osds":8,"osd_up_since":1696556465,"num_in_osds":8,"osd_in_since":1696556456,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":105}],"num_pgs":105,"num_pools":4,"num_objects":23,"data_bytes":592674,"bytes_used":2438234112,"bytes_avail":770655879168,"bytes_total":773094113280,"write_bytes_sec":2218,"read_op_per_sec":0,"write_op_per_sec":7},"fsmap":{"epoch":7,"id":1,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":1,"rank":0,"name":"d","status":"up:active","gid":4305}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2023-10-06T01:40:54.475351+0000","services":{}},"progress_events":{}} 2023-10-06T01:41:18.657 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.656+0000 7fec188de700 1 -- 172.21.15.27:0/3935661762 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7febf4066f60 msgr2=0x7febf4069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:18.657 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.656+0000 7fec188de700 1 --2- 172.21.15.27:0/3935661762 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7febf4066f60 0x7febf4069410 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fec04006f00 tx=0x7fec040091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:18.657 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.656+0000 7fec188de700 1 -- 172.21.15.27:0/3935661762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec14169ed0 msgr2=0x7fec1406c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:18.657 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.656+0000 7fec188de700 1 --2- 172.21.15.27:0/3935661762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec14169ed0 0x7fec1406c310 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fec0c00c6a0 tx=0x7fec0c015000 comp rx=0 tx=0).stop 2023-10-06T01:41:18.657 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.656+0000 7fec188de700 1 -- 172.21.15.27:0/3935661762 shutdown_connections 2023-10-06T01:41:18.657 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.656+0000 7fec188de700 1 --2- 172.21.15.27:0/3935661762 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7febf4066f60 0x7febf4069410 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:18.657 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.656+0000 7fec188de700 1 --2- 172.21.15.27:0/3935661762 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec14169ed0 0x7fec1406c310 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:18.658 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.656+0000 7fec188de700 1 --2- 172.21.15.27:0/3935661762 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7febfc004d50 0x7fec14141250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:18.658 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.656+0000 7fec188de700 1 --2- 172.21.15.27:0/3935661762 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fec140c6fa0 0x7fec14169990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:18.658 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.656+0000 7fec188de700 1 -- 172.21.15.27:0/3935661762 >> 172.21.15.27:0/3935661762 conn(0x7fec140ce4d0 msgr2=0x7fec140b38d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:18.658 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.656+0000 7fec188de700 1 -- 172.21.15.27:0/3935661762 shutdown_connections 2023-10-06T01:41:18.658 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.656+0000 7fec188de700 1 -- 172.21.15.27:0/3935661762 wait complete. 2023-10-06T01:41:18.669 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2023-10-06T01:41:18.669 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-06T01:41:18.787 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.786+0000 7f49bb2d1700 1 Processor -- start 2023-10-06T01:41:18.787 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.787+0000 7f49bb2d1700 1 -- start start 2023-10-06T01:41:18.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.787+0000 7f49bb2d1700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f49b40b5600 0x7f49b40d0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:18.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.787+0000 7f49bb2d1700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49b40b2ad0 con 0x7f49b415d930 2023-10-06T01:41:18.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.788+0000 7f49bb2d1700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49b40b4160 con 0x7f49b415d560 2023-10-06T01:41:18.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.788+0000 7f49bb2d1700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49b40b2230 con 0x7f49b40b5600 2023-10-06T01:41:18.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.788+0000 7f49b886c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f49b415d930 0x7f49b41591b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46208/0 (socket says 172.21.15.27:46208) 2023-10-06T01:41:18.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.788+0000 7f49b886c700 1 -- 172.21.15.27:0/1599161165 learned_addr learned my addr 172.21.15.27:0/1599161165 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:18.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.788+0000 7f49b986e700 1 --2- 172.21.15.27:0/1599161165 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f49b40b5600 0x7f49b40d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:18.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.788+0000 7f49b3fff700 1 -- 172.21.15.27:0/1599161165 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1874502989 0 0) 0x7f49b40b2ad0 con 0x7f49b415d930 2023-10-06T01:41:18.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.788+0000 7f49b3fff700 1 -- 172.21.15.27:0/1599161165 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f499c003570 con 0x7f49b415d930 2023-10-06T01:41:18.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.788+0000 7f49b3fff700 1 -- 172.21.15.27:0/1599161165 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2821163529 0 0) 0x7f49b40b4160 con 0x7f49b415d560 2023-10-06T01:41:18.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.788+0000 7f49b3fff700 1 -- 172.21.15.27:0/1599161165 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49b40b2ad0 con 0x7f49b415d560 2023-10-06T01:41:18.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.788+0000 7f49b986e700 1 -- 172.21.15.27:0/1599161165 >> v1:172.21.15.124:6789/0 conn(0x7f49b415d560 legacy=0x7f49b40b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:18.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.788+0000 7f49b986e700 1 -- 172.21.15.27:0/1599161165 >> v1:172.21.15.27:6789/0 conn(0x7f49b415d930 legacy=0x7f49b41591b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:18.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.788+0000 7f49b986e700 1 -- 172.21.15.27:0/1599161165 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b40b2510 con 0x7f49b40b5600 2023-10-06T01:41:18.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.788+0000 7f49b986e700 1 --2- 172.21.15.27:0/1599161165 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f49b40b5600 0x7f49b40d0030 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f49a4009910 tx=0x7f49a40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17170ede79468127 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:18.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.788+0000 7f49b3fff700 1 -- 172.21.15.27:0/1599161165 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a4011630 con 0x7f49b40b5600 2023-10-06T01:41:18.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.789+0000 7f49b3fff700 1 -- 172.21.15.27:0/1599161165 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a4011790 con 0x7f49b40b5600 2023-10-06T01:41:18.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.789+0000 7f49b3fff700 1 -- 172.21.15.27:0/1599161165 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a4011b30 con 0x7f49b40b5600 2023-10-06T01:41:18.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.789+0000 7f49bb2d1700 1 -- 172.21.15.27:0/1599161165 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f49b40b5600 msgr2=0x7f49b40d0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:18.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.789+0000 7f49bb2d1700 1 --2- 172.21.15.27:0/1599161165 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f49b40b5600 0x7f49b40d0030 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f49a4009910 tx=0x7f49a40091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:18.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.789+0000 7f49bb2d1700 1 -- 172.21.15.27:0/1599161165 shutdown_connections 2023-10-06T01:41:18.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.789+0000 7f49bb2d1700 1 --2- 172.21.15.27:0/1599161165 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f49b40b5600 0x7f49b40d0030 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:18.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.789+0000 7f49bb2d1700 1 -- 172.21.15.27:0/1599161165 >> 172.21.15.27:0/1599161165 conn(0x7f49b40d1180 msgr2=0x7f49b4069640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:18.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.789+0000 7f49bb2d1700 1 -- 172.21.15.27:0/1599161165 shutdown_connections 2023-10-06T01:41:18.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.789+0000 7f49bb2d1700 1 -- 172.21.15.27:0/1599161165 wait complete. 2023-10-06T01:41:18.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.790+0000 7f49bb2d1700 1 Processor -- start 2023-10-06T01:41:18.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.790+0000 7f49bb2d1700 1 -- start start 2023-10-06T01:41:18.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.791+0000 7f49bb2d1700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f49b40b5600 0x7f49b4153050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:18.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.791+0000 7f49bb2d1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49b415d560 0x7f49b414c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:18.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.791+0000 7f49bb2d1700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f49b415d930 0x7f49b4151780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:18.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.791+0000 7f49bb2d1700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b4152bc0 con 0x7f49b415d560 2023-10-06T01:41:18.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.791+0000 7f49bb2d1700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b414cf20 con 0x7f49b40b5600 2023-10-06T01:41:18.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.791+0000 7f49bb2d1700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49b4151fb0 con 0x7f49b415d930 2023-10-06T01:41:18.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.791+0000 7f49b886c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49b415d560 0x7f49b414c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:18.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.791+0000 7f49b886c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49b415d560 0x7f49b414c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55816/0 (socket says 172.21.15.27:55816) 2023-10-06T01:41:18.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.791+0000 7f49b886c700 1 -- 172.21.15.27:0/1644020004 learned_addr learned my addr 172.21.15.27:0/1644020004 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:18.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.791+0000 7f49b906d700 1 --2- 172.21.15.27:0/1644020004 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f49b40b5600 0x7f49b4153050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:18.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.791+0000 7f49b986e700 1 --2- 172.21.15.27:0/1644020004 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f49b415d930 0x7f49b4151780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:18.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.792+0000 7f49b886c700 1 -- 172.21.15.27:0/1644020004 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f49b415d930 msgr2=0x7f49b4151780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:18.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.792+0000 7f49b886c700 1 --2- 172.21.15.27:0/1644020004 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f49b415d930 0x7f49b4151780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:18.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.792+0000 7f49b886c700 1 -- 172.21.15.27:0/1644020004 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f49b40b5600 msgr2=0x7f49b4153050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:18.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.792+0000 7f49b886c700 1 --2- 172.21.15.27:0/1644020004 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f49b40b5600 0x7f49b4153050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:18.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.792+0000 7f49b886c700 1 -- 172.21.15.27:0/1644020004 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a4009420 con 0x7f49b415d560 2023-10-06T01:41:18.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.792+0000 7f49b986e700 1 --2- 172.21.15.27:0/1644020004 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f49b415d930 0x7f49b4151780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:18.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.792+0000 7f49b906d700 1 --2- 172.21.15.27:0/1644020004 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f49b40b5600 0x7f49b4153050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:18.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.792+0000 7f49b886c700 1 --2- 172.21.15.27:0/1644020004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49b415d560 0x7f49b414c9e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f49a800c790 tx=0x7f49a8007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:18.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.792+0000 7f49b17fa700 1 -- 172.21.15.27:0/1644020004 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a8005b30 con 0x7f49b415d560 2023-10-06T01:41:18.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.792+0000 7f49b17fa700 1 -- 172.21.15.27:0/1644020004 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a8005c90 con 0x7f49b415d560 2023-10-06T01:41:18.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.792+0000 7f49bb2d1700 1 -- 172.21.15.27:0/1644020004 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49b4152290 con 0x7f49b415d560 2023-10-06T01:41:18.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.792+0000 7f49b17fa700 1 -- 172.21.15.27:0/1644020004 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a8007300 con 0x7f49b415d560 2023-10-06T01:41:18.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.792+0000 7f49bb2d1700 1 -- 172.21.15.27:0/1644020004 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49b4152850 con 0x7f49b415d560 2023-10-06T01:41:18.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.793+0000 7f49bb2d1700 1 -- 172.21.15.27:0/1644020004 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49b4152d30 con 0x7f49b415d560 2023-10-06T01:41:18.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.793+0000 7f49b17fa700 1 -- 172.21.15.27:0/1644020004 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f49a801b3b0 con 0x7f49b415d560 2023-10-06T01:41:18.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.794+0000 7f49b17fa700 1 --2- 172.21.15.27:0/1644020004 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f49a0066f60 0x7f49a0069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:18.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.794+0000 7f49b17fa700 1 -- 172.21.15.27:0/1644020004 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f49a8083c90 con 0x7f49b415d560 2023-10-06T01:41:18.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.794+0000 7f49b906d700 1 --2- 172.21.15.27:0/1644020004 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f49a0066f60 0x7f49a0069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:18.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.794+0000 7f49b906d700 1 --2- 172.21.15.27:0/1644020004 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f49a0066f60 0x7f49a0069410 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f49ac004040 tx=0x7f49ac009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:18.798 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.798+0000 7f49b17fa700 1 -- 172.21.15.27:0/1644020004 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f49a8050da0 con 0x7f49b415d560 2023-10-06T01:41:18.985 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.985+0000 7f49bb2d1700 1 -- 172.21.15.27:0/1644020004 --> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f49b4064050 con 0x7f49a0066f60 2023-10-06T01:41:18.993 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.993+0000 7f49b17fa700 1 -- 172.21.15.27:0/1644020004 <== mgr.4100 v2:172.21.15.124:6832/90278 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286886 (secure 0 0 0) 0x7f49b4064050 con 0x7f49a0066f60 2023-10-06T01:41:18.996 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:18.999 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.996+0000 7f49bb2d1700 1 -- 172.21.15.27:0/1644020004 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f49a0066f60 msgr2=0x7f49a0069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:18.999 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.996+0000 7f49bb2d1700 1 --2- 172.21.15.27:0/1644020004 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f49a0066f60 0x7f49a0069410 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f49ac004040 tx=0x7f49ac009040 comp rx=0 tx=0).stop 2023-10-06T01:41:18.999 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.996+0000 7f49bb2d1700 1 -- 172.21.15.27:0/1644020004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49b415d560 msgr2=0x7f49b414c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:18.999 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.996+0000 7f49bb2d1700 1 --2- 172.21.15.27:0/1644020004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49b415d560 0x7f49b414c9e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f49a800c790 tx=0x7f49a8007ea0 comp rx=0 tx=0).stop 2023-10-06T01:41:18.999 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.996+0000 7f49bb2d1700 1 -- 172.21.15.27:0/1644020004 shutdown_connections 2023-10-06T01:41:18.999 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.996+0000 7f49bb2d1700 1 --2- 172.21.15.27:0/1644020004 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f49a0066f60 0x7f49a0069410 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:18.999 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.996+0000 7f49bb2d1700 1 --2- 172.21.15.27:0/1644020004 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f49b415d930 0x7f49b4151780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:18.999 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.996+0000 7f49bb2d1700 1 --2- 172.21.15.27:0/1644020004 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f49b40b5600 0x7f49b4153050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:19.000 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.996+0000 7f49bb2d1700 1 --2- 172.21.15.27:0/1644020004 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f49b415d560 0x7f49b414c9e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:19.000 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.996+0000 7f49bb2d1700 1 -- 172.21.15.27:0/1644020004 >> 172.21.15.27:0/1644020004 conn(0x7f49b40d1180 msgr2=0x7f49b4068250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:19.000 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.996+0000 7f49bb2d1700 1 -- 172.21.15.27:0/1644020004 shutdown_connections 2023-10-06T01:41:19.000 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:18.996+0000 7f49bb2d1700 1 -- 172.21.15.27:0/1644020004 wait complete. 2023-10-06T01:41:19.000 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-06T01:41:19.008 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2023-10-06T01:41:18.039083+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":112,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2381088,"kb_used_data":2784,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752593632,"statfs":{"total":773094113280,"available":770655879168,"internally_reserved":0,"allocated":2850816,"data_stored":1760126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001420"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.349688+0000","last_change":"2023-10-06T01:41:13.346440+0000","last_active":"2023-10-06T01:41:13.349688+0000","last_peered":"2023-10-06T01:41:13.349688+0000","last_clean":"2023-10-06T01:41:13.349688+0000","last_became_active":"2023-10-06T01:41:13.343345+0000","last_became_peered":"2023-10-06T01:41:13.343345+0000","last_unstale":"2023-10-06T01:41:13.349688+0000","last_undegraded":"2023-10-06T01:41:13.349688+0000","last_fullsized":"2023-10-06T01:41:13.349688+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:15:41.848935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354765+0000","last_change":"2023-10-06T01:41:13.348847+0000","last_active":"2023-10-06T01:41:13.354765+0000","last_peered":"2023-10-06T01:41:13.354765+0000","last_clean":"2023-10-06T01:41:13.354765+0000","last_became_active":"2023-10-06T01:41:13.348585+0000","last_became_peered":"2023-10-06T01:41:13.348585+0000","last_unstale":"2023-10-06T01:41:13.354765+0000","last_undegraded":"2023-10-06T01:41:13.354765+0000","last_fullsized":"2023-10-06T01:41:13.354765+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:31:20.840764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354463+0000","last_change":"2023-10-06T01:41:13.345122+0000","last_active":"2023-10-06T01:41:13.354463+0000","last_peered":"2023-10-06T01:41:13.354463+0000","last_clean":"2023-10-06T01:41:13.354463+0000","last_became_active":"2023-10-06T01:41:13.344752+0000","last_became_peered":"2023-10-06T01:41:13.344752+0000","last_unstale":"2023-10-06T01:41:13.354463+0000","last_undegraded":"2023-10-06T01:41:13.354463+0000","last_fullsized":"2023-10-06T01:41:13.354463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:42:56.041827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.342734+0000","last_change":"2023-10-06T01:41:13.340589+0000","last_active":"2023-10-06T01:41:13.342734+0000","last_peered":"2023-10-06T01:41:13.342734+0000","last_clean":"2023-10-06T01:41:13.342734+0000","last_became_active":"2023-10-06T01:41:13.340339+0000","last_became_peered":"2023-10-06T01:41:13.340339+0000","last_unstale":"2023-10-06T01:41:13.342734+0000","last_undegraded":"2023-10-06T01:41:13.342734+0000","last_fullsized":"2023-10-06T01:41:13.342734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:32:17.938309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.342002+0000","last_change":"2023-10-06T01:41:13.342002+0000","last_active":"2023-10-06T01:41:13.342002+0000","last_peered":"2023-10-06T01:41:13.342002+0000","last_clean":"2023-10-06T01:41:13.342002+0000","last_became_active":"2023-10-06T01:41:13.341403+0000","last_became_peered":"2023-10-06T01:41:13.341403+0000","last_unstale":"2023-10-06T01:41:13.342002+0000","last_undegraded":"2023-10-06T01:41:13.342002+0000","last_fullsized":"2023-10-06T01:41:13.342002+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:34:55.375989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354785+0000","last_change":"2023-10-06T01:41:13.345138+0000","last_active":"2023-10-06T01:41:13.354785+0000","last_peered":"2023-10-06T01:41:13.354785+0000","last_clean":"2023-10-06T01:41:13.354785+0000","last_became_active":"2023-10-06T01:41:13.344810+0000","last_became_peered":"2023-10-06T01:41:13.344810+0000","last_unstale":"2023-10-06T01:41:13.354785+0000","last_undegraded":"2023-10-06T01:41:13.354785+0000","last_fullsized":"2023-10-06T01:41:13.354785+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:17:59.027125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356675+0000","last_change":"2023-10-06T01:41:13.352141+0000","last_active":"2023-10-06T01:41:13.356675+0000","last_peered":"2023-10-06T01:41:13.356675+0000","last_clean":"2023-10-06T01:41:13.356675+0000","last_became_active":"2023-10-06T01:41:13.351628+0000","last_became_peered":"2023-10-06T01:41:13.351628+0000","last_unstale":"2023-10-06T01:41:13.356675+0000","last_undegraded":"2023-10-06T01:41:13.356675+0000","last_fullsized":"2023-10-06T01:41:13.356675+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:03:37.385158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354278+0000","last_change":"2023-10-06T01:41:13.345960+0000","last_active":"2023-10-06T01:41:13.354278+0000","last_peered":"2023-10-06T01:41:13.354278+0000","last_clean":"2023-10-06T01:41:13.354278+0000","last_became_active":"2023-10-06T01:41:13.345702+0000","last_became_peered":"2023-10-06T01:41:13.345702+0000","last_unstale":"2023-10-06T01:41:13.354278+0000","last_undegraded":"2023-10-06T01:41:13.354278+0000","last_fullsized":"2023-10-06T01:41:13.354278+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:03:37.385158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.343792+0000","last_change":"2023-10-06T01:41:13.343792+0000","last_active":"2023-10-06T01:41:13.343792+0000","last_peered":"2023-10-06T01:41:13.343792+0000","last_clean":"2023-10-06T01:41:13.343792+0000","last_became_active":"2023-10-06T01:41:13.343432+0000","last_became_peered":"2023-10-06T01:41:13.343432+0000","last_unstale":"2023-10-06T01:41:13.343792+0000","last_undegraded":"2023-10-06T01:41:13.343792+0000","last_fullsized":"2023-10-06T01:41:13.343792+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:52:48.286318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354747+0000","last_change":"2023-10-06T01:41:13.347399+0000","last_active":"2023-10-06T01:41:13.354747+0000","last_peered":"2023-10-06T01:41:13.354747+0000","last_clean":"2023-10-06T01:41:13.354747+0000","last_became_active":"2023-10-06T01:41:13.347136+0000","last_became_peered":"2023-10-06T01:41:13.347136+0000","last_unstale":"2023-10-06T01:41:13.354747+0000","last_undegraded":"2023-10-06T01:41:13.354747+0000","last_fullsized":"2023-10-06T01:41:13.354747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:42:23.199959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354772+0000","last_change":"2023-10-06T01:41:13.350059+0000","last_active":"2023-10-06T01:41:13.354772+0000","last_peered":"2023-10-06T01:41:13.354772+0000","last_clean":"2023-10-06T01:41:13.354772+0000","last_became_active":"2023-10-06T01:41:13.349673+0000","last_became_peered":"2023-10-06T01:41:13.349673+0000","last_unstale":"2023-10-06T01:41:13.354772+0000","last_undegraded":"2023-10-06T01:41:13.354772+0000","last_fullsized":"2023-10-06T01:41:13.354772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:12:57.315612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351538+0000","last_change":"2023-10-06T01:41:13.348001+0000","last_active":"2023-10-06T01:41:13.351538+0000","last_peered":"2023-10-06T01:41:13.351538+0000","last_clean":"2023-10-06T01:41:13.351538+0000","last_became_active":"2023-10-06T01:41:13.347719+0000","last_became_peered":"2023-10-06T01:41:13.347719+0000","last_unstale":"2023-10-06T01:41:13.351538+0000","last_undegraded":"2023-10-06T01:41:13.351538+0000","last_fullsized":"2023-10-06T01:41:13.351538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:37:53.150671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.355181+0000","last_change":"2023-10-06T01:41:13.352875+0000","last_active":"2023-10-06T01:41:13.355181+0000","last_peered":"2023-10-06T01:41:13.355181+0000","last_clean":"2023-10-06T01:41:13.355181+0000","last_became_active":"2023-10-06T01:41:13.352589+0000","last_became_peered":"2023-10-06T01:41:13.352589+0000","last_unstale":"2023-10-06T01:41:13.355181+0000","last_undegraded":"2023-10-06T01:41:13.355181+0000","last_fullsized":"2023-10-06T01:41:13.355181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:49:55.665032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354387+0000","last_change":"2023-10-06T01:41:13.348362+0000","last_active":"2023-10-06T01:41:13.354387+0000","last_peered":"2023-10-06T01:41:13.354387+0000","last_clean":"2023-10-06T01:41:13.354387+0000","last_became_active":"2023-10-06T01:41:13.348181+0000","last_became_peered":"2023-10-06T01:41:13.348181+0000","last_unstale":"2023-10-06T01:41:13.354387+0000","last_undegraded":"2023-10-06T01:41:13.354387+0000","last_fullsized":"2023-10-06T01:41:13.354387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:27:54.074728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356697+0000","last_change":"2023-10-06T01:41:13.352048+0000","last_active":"2023-10-06T01:41:13.356697+0000","last_peered":"2023-10-06T01:41:13.356697+0000","last_clean":"2023-10-06T01:41:13.356697+0000","last_became_active":"2023-10-06T01:41:13.351640+0000","last_became_peered":"2023-10-06T01:41:13.351640+0000","last_unstale":"2023-10-06T01:41:13.356697+0000","last_undegraded":"2023-10-06T01:41:13.356697+0000","last_fullsized":"2023-10-06T01:41:13.356697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:42:23.199959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354602+0000","last_change":"2023-10-06T01:41:13.345677+0000","last_active":"2023-10-06T01:41:13.354602+0000","last_peered":"2023-10-06T01:41:13.354602+0000","last_clean":"2023-10-06T01:41:13.354602+0000","last_became_active":"2023-10-06T01:41:13.345257+0000","last_became_peered":"2023-10-06T01:41:13.345257+0000","last_unstale":"2023-10-06T01:41:13.354602+0000","last_undegraded":"2023-10-06T01:41:13.354602+0000","last_fullsized":"2023-10-06T01:41:13.354602+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:27:07.040029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.344490+0000","last_change":"2023-10-06T01:41:13.344490+0000","last_active":"2023-10-06T01:41:13.344490+0000","last_peered":"2023-10-06T01:41:13.344490+0000","last_clean":"2023-10-06T01:41:13.344490+0000","last_became_active":"2023-10-06T01:41:13.344220+0000","last_became_peered":"2023-10-06T01:41:13.344220+0000","last_unstale":"2023-10-06T01:41:13.344490+0000","last_undegraded":"2023-10-06T01:41:13.344490+0000","last_fullsized":"2023-10-06T01:41:13.344490+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T01:51:57.229301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354756+0000","last_change":"2023-10-06T01:41:13.350575+0000","last_active":"2023-10-06T01:41:13.354756+0000","last_peered":"2023-10-06T01:41:13.354756+0000","last_clean":"2023-10-06T01:41:13.354756+0000","last_became_active":"2023-10-06T01:41:13.350311+0000","last_became_peered":"2023-10-06T01:41:13.350311+0000","last_unstale":"2023-10-06T01:41:13.354756+0000","last_undegraded":"2023-10-06T01:41:13.354756+0000","last_fullsized":"2023-10-06T01:41:13.354756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:39:03.091509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351594+0000","last_change":"2023-10-06T01:41:13.347386+0000","last_active":"2023-10-06T01:41:13.351594+0000","last_peered":"2023-10-06T01:41:13.351594+0000","last_clean":"2023-10-06T01:41:13.351594+0000","last_became_active":"2023-10-06T01:41:13.347101+0000","last_became_peered":"2023-10-06T01:41:13.347101+0000","last_unstale":"2023-10-06T01:41:13.351594+0000","last_undegraded":"2023-10-06T01:41:13.351594+0000","last_fullsized":"2023-10-06T01:41:13.351594+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:13:15.011521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351504+0000","last_change":"2023-10-06T01:41:13.342640+0000","last_active":"2023-10-06T01:41:13.351504+0000","last_peered":"2023-10-06T01:41:13.351504+0000","last_clean":"2023-10-06T01:41:13.351504+0000","last_became_active":"2023-10-06T01:41:13.342157+0000","last_became_peered":"2023-10-06T01:41:13.342157+0000","last_unstale":"2023-10-06T01:41:13.351504+0000","last_undegraded":"2023-10-06T01:41:13.351504+0000","last_fullsized":"2023-10-06T01:41:13.351504+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:07:52.521701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356729+0000","last_change":"2023-10-06T01:41:13.352582+0000","last_active":"2023-10-06T01:41:13.356729+0000","last_peered":"2023-10-06T01:41:13.356729+0000","last_clean":"2023-10-06T01:41:13.356729+0000","last_became_active":"2023-10-06T01:41:13.352281+0000","last_became_peered":"2023-10-06T01:41:13.352281+0000","last_unstale":"2023-10-06T01:41:13.356729+0000","last_undegraded":"2023-10-06T01:41:13.356729+0000","last_fullsized":"2023-10-06T01:41:13.356729+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:31:20.840764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.345324+0000","last_change":"2023-10-06T01:41:13.345324+0000","last_active":"2023-10-06T01:41:13.345324+0000","last_peered":"2023-10-06T01:41:13.345324+0000","last_clean":"2023-10-06T01:41:13.345324+0000","last_became_active":"2023-10-06T01:41:13.344997+0000","last_became_peered":"2023-10-06T01:41:13.344997+0000","last_unstale":"2023-10-06T01:41:13.345324+0000","last_undegraded":"2023-10-06T01:41:13.345324+0000","last_fullsized":"2023-10-06T01:41:13.345324+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:27:54.074728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.355108+0000","last_change":"2023-10-06T01:41:13.355108+0000","last_active":"2023-10-06T01:41:13.355108+0000","last_peered":"2023-10-06T01:41:13.355108+0000","last_clean":"2023-10-06T01:41:13.355108+0000","last_became_active":"2023-10-06T01:41:13.354621+0000","last_became_peered":"2023-10-06T01:41:13.354621+0000","last_unstale":"2023-10-06T01:41:13.355108+0000","last_undegraded":"2023-10-06T01:41:13.355108+0000","last_fullsized":"2023-10-06T01:41:13.355108+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:34:55.375989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356702+0000","last_change":"2023-10-06T01:41:13.352301+0000","last_active":"2023-10-06T01:41:13.356702+0000","last_peered":"2023-10-06T01:41:13.356702+0000","last_clean":"2023-10-06T01:41:13.356702+0000","last_became_active":"2023-10-06T01:41:13.351858+0000","last_became_peered":"2023-10-06T01:41:13.351858+0000","last_unstale":"2023-10-06T01:41:13.356702+0000","last_undegraded":"2023-10-06T01:41:13.356702+0000","last_fullsized":"2023-10-06T01:41:13.356702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:42:46.214805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.348956+0000","last_change":"2023-10-06T01:41:13.344094+0000","last_active":"2023-10-06T01:41:13.348956+0000","last_peered":"2023-10-06T01:41:13.348956+0000","last_clean":"2023-10-06T01:41:13.348956+0000","last_became_active":"2023-10-06T01:41:13.343796+0000","last_became_peered":"2023-10-06T01:41:13.343796+0000","last_unstale":"2023-10-06T01:41:13.348956+0000","last_undegraded":"2023-10-06T01:41:13.348956+0000","last_fullsized":"2023-10-06T01:41:13.348956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:21:30.451862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.355700+0000","last_change":"2023-10-06T01:41:13.350352+0000","last_active":"2023-10-06T01:41:13.355700+0000","last_peered":"2023-10-06T01:41:13.355700+0000","last_clean":"2023-10-06T01:41:13.355700+0000","last_became_active":"2023-10-06T01:41:13.350133+0000","last_became_peered":"2023-10-06T01:41:13.350133+0000","last_unstale":"2023-10-06T01:41:13.355700+0000","last_undegraded":"2023-10-06T01:41:13.355700+0000","last_fullsized":"2023-10-06T01:41:13.355700+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:05:37.961571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354695+0000","last_change":"2023-10-06T01:41:13.353241+0000","last_active":"2023-10-06T01:41:13.354695+0000","last_peered":"2023-10-06T01:41:13.354695+0000","last_clean":"2023-10-06T01:41:13.354695+0000","last_became_active":"2023-10-06T01:41:13.352824+0000","last_became_peered":"2023-10-06T01:41:13.352824+0000","last_unstale":"2023-10-06T01:41:13.354695+0000","last_undegraded":"2023-10-06T01:41:13.354695+0000","last_fullsized":"2023-10-06T01:41:13.354695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:52:48.286318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"19'1","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.348819+0000","last_change":"2023-10-06T01:41:12.339675+0000","last_active":"2023-10-06T01:41:13.348819+0000","last_peered":"2023-10-06T01:41:13.348819+0000","last_clean":"2023-10-06T01:41:13.348819+0000","last_became_active":"2023-10-06T01:41:12.339258+0000","last_became_peered":"2023-10-06T01:41:12.339258+0000","last_unstale":"2023-10-06T01:41:13.348819+0000","last_undegraded":"2023-10-06T01:41:13.348819+0000","last_fullsized":"2023-10-06T01:41:13.348819+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:49:54.658088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352444+0000","last_change":"2023-10-06T01:41:13.352444+0000","last_active":"2023-10-06T01:41:13.352444+0000","last_peered":"2023-10-06T01:41:13.352444+0000","last_clean":"2023-10-06T01:41:13.352444+0000","last_became_active":"2023-10-06T01:41:13.352134+0000","last_became_peered":"2023-10-06T01:41:13.352134+0000","last_unstale":"2023-10-06T01:41:13.352444+0000","last_undegraded":"2023-10-06T01:41:13.352444+0000","last_fullsized":"2023-10-06T01:41:13.352444+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:11:26.794025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.336930+0000","last_change":"2023-10-06T01:41:12.347654+0000","last_active":"2023-10-06T01:41:13.336930+0000","last_peered":"2023-10-06T01:41:13.336930+0000","last_clean":"2023-10-06T01:41:13.336930+0000","last_became_active":"2023-10-06T01:41:12.347386+0000","last_became_peered":"2023-10-06T01:41:12.347386+0000","last_unstale":"2023-10-06T01:41:13.336930+0000","last_undegraded":"2023-10-06T01:41:13.336930+0000","last_fullsized":"2023-10-06T01:41:13.336930+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:07:30.529679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352350+0000","last_change":"2023-10-06T01:41:13.352350+0000","last_active":"2023-10-06T01:41:13.352350+0000","last_peered":"2023-10-06T01:41:13.352350+0000","last_clean":"2023-10-06T01:41:13.352350+0000","last_became_active":"2023-10-06T01:41:13.352091+0000","last_became_peered":"2023-10-06T01:41:13.352091+0000","last_unstale":"2023-10-06T01:41:13.352350+0000","last_undegraded":"2023-10-06T01:41:13.352350+0000","last_fullsized":"2023-10-06T01:41:13.352350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:31:20.840764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-10-06T01:41:12.349520+0000","last_change":"2023-10-06T01:41:12.343086+0000","last_active":"2023-10-06T01:41:12.349520+0000","last_peered":"2023-10-06T01:41:12.349520+0000","last_clean":"2023-10-06T01:41:12.349520+0000","last_became_active":"2023-10-06T01:41:12.342595+0000","last_became_peered":"2023-10-06T01:41:12.342595+0000","last_unstale":"2023-10-06T01:41:12.349520+0000","last_undegraded":"2023-10-06T01:41:12.349520+0000","last_fullsized":"2023-10-06T01:41:12.349520+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:15:40.841990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351933+0000","last_change":"2023-10-06T01:41:13.347027+0000","last_active":"2023-10-06T01:41:13.351933+0000","last_peered":"2023-10-06T01:41:13.351933+0000","last_clean":"2023-10-06T01:41:13.351933+0000","last_became_active":"2023-10-06T01:41:13.346469+0000","last_became_peered":"2023-10-06T01:41:13.346469+0000","last_unstale":"2023-10-06T01:41:13.351933+0000","last_undegraded":"2023-10-06T01:41:13.351933+0000","last_fullsized":"2023-10-06T01:41:13.351933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:37:14.679660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-10-06T01:41:12.348510+0000","last_change":"2023-10-06T01:41:12.341922+0000","last_active":"2023-10-06T01:41:12.348510+0000","last_peered":"2023-10-06T01:41:12.348510+0000","last_clean":"2023-10-06T01:41:12.348510+0000","last_became_active":"2023-10-06T01:41:12.341543+0000","last_became_peered":"2023-10-06T01:41:12.341543+0000","last_unstale":"2023-10-06T01:41:12.348510+0000","last_undegraded":"2023-10-06T01:41:12.348510+0000","last_fullsized":"2023-10-06T01:41:12.348510+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:13:14.004577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.345274+0000","last_change":"2023-10-06T01:41:13.345274+0000","last_active":"2023-10-06T01:41:13.345274+0000","last_peered":"2023-10-06T01:41:13.345274+0000","last_clean":"2023-10-06T01:41:13.345274+0000","last_became_active":"2023-10-06T01:41:13.344992+0000","last_became_peered":"2023-10-06T01:41:13.344992+0000","last_unstale":"2023-10-06T01:41:13.345274+0000","last_undegraded":"2023-10-06T01:41:13.345274+0000","last_fullsized":"2023-10-06T01:41:13.345274+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:39:03.091509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.339176+0000","last_change":"2023-10-06T01:41:12.343531+0000","last_active":"2023-10-06T01:41:13.339176+0000","last_peered":"2023-10-06T01:41:13.339176+0000","last_clean":"2023-10-06T01:41:13.339176+0000","last_became_active":"2023-10-06T01:41:12.343256+0000","last_became_peered":"2023-10-06T01:41:12.343256+0000","last_unstale":"2023-10-06T01:41:13.339176+0000","last_undegraded":"2023-10-06T01:41:13.339176+0000","last_fullsized":"2023-10-06T01:41:13.339176+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:16:14.826869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.357361+0000","last_change":"2023-10-06T01:41:13.357361+0000","last_active":"2023-10-06T01:41:13.357361+0000","last_peered":"2023-10-06T01:41:13.357361+0000","last_clean":"2023-10-06T01:41:13.357361+0000","last_became_active":"2023-10-06T01:41:13.356980+0000","last_became_peered":"2023-10-06T01:41:13.356980+0000","last_unstale":"2023-10-06T01:41:13.357361+0000","last_undegraded":"2023-10-06T01:41:13.357361+0000","last_fullsized":"2023-10-06T01:41:13.357361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:42:51.581939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"19'3","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.355632+0000","last_change":"2023-10-06T01:41:12.350010+0000","last_active":"2023-10-06T01:41:13.355632+0000","last_peered":"2023-10-06T01:41:13.355632+0000","last_clean":"2023-10-06T01:41:13.355632+0000","last_became_active":"2023-10-06T01:41:12.349758+0000","last_became_peered":"2023-10-06T01:41:12.349758+0000","last_unstale":"2023-10-06T01:41:13.355632+0000","last_undegraded":"2023-10-06T01:41:13.355632+0000","last_fullsized":"2023-10-06T01:41:13.355632+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:12:51.734395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352993+0000","last_change":"2023-10-06T01:41:13.348953+0000","last_active":"2023-10-06T01:41:13.352993+0000","last_peered":"2023-10-06T01:41:13.352993+0000","last_clean":"2023-10-06T01:41:13.352993+0000","last_became_active":"2023-10-06T01:41:13.348683+0000","last_became_peered":"2023-10-06T01:41:13.348683+0000","last_unstale":"2023-10-06T01:41:13.352993+0000","last_undegraded":"2023-10-06T01:41:13.352993+0000","last_fullsized":"2023-10-06T01:41:13.352993+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:42:46.214805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"19'6","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.357074+0000","last_change":"2023-10-06T01:41:12.340922+0000","last_active":"2023-10-06T01:41:13.357074+0000","last_peered":"2023-10-06T01:41:13.357074+0000","last_clean":"2023-10-06T01:41:13.357074+0000","last_became_active":"2023-10-06T01:41:12.340598+0000","last_became_peered":"2023-10-06T01:41:12.340598+0000","last_unstale":"2023-10-06T01:41:13.357074+0000","last_undegraded":"2023-10-06T01:41:13.357074+0000","last_fullsized":"2023-10-06T01:41:13.357074+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:06:19.029137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351653+0000","last_change":"2023-10-06T01:41:13.343918+0000","last_active":"2023-10-06T01:41:13.351653+0000","last_peered":"2023-10-06T01:41:13.351653+0000","last_clean":"2023-10-06T01:41:13.351653+0000","last_became_active":"2023-10-06T01:41:13.343621+0000","last_became_peered":"2023-10-06T01:41:13.343621+0000","last_unstale":"2023-10-06T01:41:13.351653+0000","last_undegraded":"2023-10-06T01:41:13.351653+0000","last_fullsized":"2023-10-06T01:41:13.351653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:42:44.799267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.350439+0000","last_change":"2023-10-06T01:41:12.343541+0000","last_active":"2023-10-06T01:41:13.350439+0000","last_peered":"2023-10-06T01:41:13.350439+0000","last_clean":"2023-10-06T01:41:13.350439+0000","last_became_active":"2023-10-06T01:41:12.342645+0000","last_became_peered":"2023-10-06T01:41:12.342645+0000","last_unstale":"2023-10-06T01:41:13.350439+0000","last_undegraded":"2023-10-06T01:41:13.350439+0000","last_fullsized":"2023-10-06T01:41:13.350439+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:16:14.826869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352574+0000","last_change":"2023-10-06T01:41:13.352574+0000","last_active":"2023-10-06T01:41:13.352574+0000","last_peered":"2023-10-06T01:41:13.352574+0000","last_clean":"2023-10-06T01:41:13.352574+0000","last_became_active":"2023-10-06T01:41:13.352300+0000","last_became_peered":"2023-10-06T01:41:13.352300+0000","last_unstale":"2023-10-06T01:41:13.352574+0000","last_undegraded":"2023-10-06T01:41:13.352574+0000","last_fullsized":"2023-10-06T01:41:13.352574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:27:54.074728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-10-06T01:41:12.351884+0000","last_change":"2023-10-06T01:41:12.344679+0000","last_active":"2023-10-06T01:41:12.351884+0000","last_peered":"2023-10-06T01:41:12.351884+0000","last_clean":"2023-10-06T01:41:12.351884+0000","last_became_active":"2023-10-06T01:41:12.344430+0000","last_became_peered":"2023-10-06T01:41:12.344430+0000","last_unstale":"2023-10-06T01:41:12.351884+0000","last_undegraded":"2023-10-06T01:41:12.351884+0000","last_fullsized":"2023-10-06T01:41:12.351884+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:27:01.582307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.360786+0000","last_change":"2023-10-06T01:41:13.357148+0000","last_active":"2023-10-06T01:41:13.360786+0000","last_peered":"2023-10-06T01:41:13.360786+0000","last_clean":"2023-10-06T01:41:13.360786+0000","last_became_active":"2023-10-06T01:41:13.356843+0000","last_became_peered":"2023-10-06T01:41:13.356843+0000","last_unstale":"2023-10-06T01:41:13.360786+0000","last_undegraded":"2023-10-06T01:41:13.360786+0000","last_fullsized":"2023-10-06T01:41:13.360786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T01:51:57.229301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.334167+0000","last_change":"2023-10-06T01:41:12.344068+0000","last_active":"2023-10-06T01:41:13.334167+0000","last_peered":"2023-10-06T01:41:13.334167+0000","last_clean":"2023-10-06T01:41:13.334167+0000","last_became_active":"2023-10-06T01:41:12.342562+0000","last_became_peered":"2023-10-06T01:41:12.342562+0000","last_unstale":"2023-10-06T01:41:13.334167+0000","last_undegraded":"2023-10-06T01:41:13.334167+0000","last_fullsized":"2023-10-06T01:41:13.334167+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:37:13.672716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.343508+0000","last_change":"2023-10-06T01:41:13.343508+0000","last_active":"2023-10-06T01:41:13.343508+0000","last_peered":"2023-10-06T01:41:13.343508+0000","last_clean":"2023-10-06T01:41:13.343508+0000","last_became_active":"2023-10-06T01:41:13.343249+0000","last_became_peered":"2023-10-06T01:41:13.343249+0000","last_unstale":"2023-10-06T01:41:13.343508+0000","last_undegraded":"2023-10-06T01:41:13.343508+0000","last_fullsized":"2023-10-06T01:41:13.343508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:03:37.385158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"19'8","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.347028+0000","last_change":"2023-10-06T01:41:12.342949+0000","last_active":"2023-10-06T01:41:13.347028+0000","last_peered":"2023-10-06T01:41:13.347028+0000","last_clean":"2023-10-06T01:41:13.347028+0000","last_became_active":"2023-10-06T01:41:12.342687+0000","last_became_peered":"2023-10-06T01:41:12.342687+0000","last_unstale":"2023-10-06T01:41:13.347028+0000","last_undegraded":"2023-10-06T01:41:13.347028+0000","last_fullsized":"2023-10-06T01:41:13.347028+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:37:52.143727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354861+0000","last_change":"2023-10-06T01:41:13.354861+0000","last_active":"2023-10-06T01:41:13.354861+0000","last_peered":"2023-10-06T01:41:13.354861+0000","last_clean":"2023-10-06T01:41:13.354861+0000","last_became_active":"2023-10-06T01:41:13.354190+0000","last_became_peered":"2023-10-06T01:41:13.354190+0000","last_unstale":"2023-10-06T01:41:13.354861+0000","last_undegraded":"2023-10-06T01:41:13.354861+0000","last_fullsized":"2023-10-06T01:41:13.354861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:45:07.571069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"19'1","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.349986+0000","last_change":"2023-10-06T01:41:12.344048+0000","last_active":"2023-10-06T01:41:13.349986+0000","last_peered":"2023-10-06T01:41:13.349986+0000","last_clean":"2023-10-06T01:41:13.349986+0000","last_became_active":"2023-10-06T01:41:12.343532+0000","last_became_peered":"2023-10-06T01:41:12.343532+0000","last_unstale":"2023-10-06T01:41:13.349986+0000","last_undegraded":"2023-10-06T01:41:13.349986+0000","last_fullsized":"2023-10-06T01:41:13.349986+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:50:44.552768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354457+0000","last_change":"2023-10-06T01:41:13.351276+0000","last_active":"2023-10-06T01:41:13.354457+0000","last_peered":"2023-10-06T01:41:13.354457+0000","last_clean":"2023-10-06T01:41:13.354457+0000","last_became_active":"2023-10-06T01:41:13.351005+0000","last_became_peered":"2023-10-06T01:41:13.351005+0000","last_unstale":"2023-10-06T01:41:13.354457+0000","last_undegraded":"2023-10-06T01:41:13.354457+0000","last_fullsized":"2023-10-06T01:41:13.354457+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:42:51.581939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356755+0000","last_change":"2023-10-06T01:41:12.344159+0000","last_active":"2023-10-06T01:41:13.356755+0000","last_peered":"2023-10-06T01:41:13.356755+0000","last_clean":"2023-10-06T01:41:13.356755+0000","last_became_active":"2023-10-06T01:41:12.343593+0000","last_became_peered":"2023-10-06T01:41:12.343593+0000","last_unstale":"2023-10-06T01:41:13.356755+0000","last_undegraded":"2023-10-06T01:41:13.356755+0000","last_fullsized":"2023-10-06T01:41:13.356755+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:27:01.582307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354635+0000","last_change":"2023-10-06T01:41:13.349120+0000","last_active":"2023-10-06T01:41:13.354635+0000","last_peered":"2023-10-06T01:41:13.354635+0000","last_clean":"2023-10-06T01:41:13.354635+0000","last_became_active":"2023-10-06T01:41:13.348850+0000","last_became_peered":"2023-10-06T01:41:13.348850+0000","last_unstale":"2023-10-06T01:41:13.354635+0000","last_undegraded":"2023-10-06T01:41:13.354635+0000","last_fullsized":"2023-10-06T01:41:13.354635+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:27:32.273170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"19'1","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.358476+0000","last_change":"2023-10-06T01:41:12.344880+0000","last_active":"2023-10-06T01:41:13.358476+0000","last_peered":"2023-10-06T01:41:13.358476+0000","last_clean":"2023-10-06T01:41:13.358476+0000","last_became_active":"2023-10-06T01:41:12.344676+0000","last_became_peered":"2023-10-06T01:41:12.344676+0000","last_unstale":"2023-10-06T01:41:13.358476+0000","last_undegraded":"2023-10-06T01:41:13.358476+0000","last_fullsized":"2023-10-06T01:41:13.358476+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:21:09.833061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.353951+0000","last_change":"2023-10-06T01:41:13.348117+0000","last_active":"2023-10-06T01:41:13.353951+0000","last_peered":"2023-10-06T01:41:13.353951+0000","last_clean":"2023-10-06T01:41:13.353951+0000","last_became_active":"2023-10-06T01:41:13.347872+0000","last_became_peered":"2023-10-06T01:41:13.347872+0000","last_unstale":"2023-10-06T01:41:13.353951+0000","last_undegraded":"2023-10-06T01:41:13.353951+0000","last_fullsized":"2023-10-06T01:41:13.353951+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:21:10.840006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.338677+0000","last_change":"2023-10-06T01:41:12.341362+0000","last_active":"2023-10-06T01:41:13.338677+0000","last_peered":"2023-10-06T01:41:13.338677+0000","last_clean":"2023-10-06T01:41:13.338677+0000","last_became_active":"2023-10-06T01:41:12.340855+0000","last_became_peered":"2023-10-06T01:41:12.340855+0000","last_unstale":"2023-10-06T01:41:13.338677+0000","last_undegraded":"2023-10-06T01:41:13.338677+0000","last_fullsized":"2023-10-06T01:41:13.338677+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:15:40.841990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.344966+0000","last_change":"2023-10-06T01:41:13.344966+0000","last_active":"2023-10-06T01:41:13.344966+0000","last_peered":"2023-10-06T01:41:13.344966+0000","last_clean":"2023-10-06T01:41:13.344966+0000","last_became_active":"2023-10-06T01:41:13.344700+0000","last_became_peered":"2023-10-06T01:41:13.344700+0000","last_unstale":"2023-10-06T01:41:13.344966+0000","last_undegraded":"2023-10-06T01:41:13.344966+0000","last_fullsized":"2023-10-06T01:41:13.344966+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:42:23.199959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"19'2","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.346245+0000","last_change":"2023-10-06T01:41:12.343169+0000","last_active":"2023-10-06T01:41:13.346245+0000","last_peered":"2023-10-06T01:41:13.346245+0000","last_clean":"2023-10-06T01:41:13.346245+0000","last_became_active":"2023-10-06T01:41:12.342474+0000","last_became_peered":"2023-10-06T01:41:12.342474+0000","last_unstale":"2023-10-06T01:41:13.346245+0000","last_undegraded":"2023-10-06T01:41:13.346245+0000","last_fullsized":"2023-10-06T01:41:13.346245+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:27:01.582307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351608+0000","last_change":"2023-10-06T01:41:13.343671+0000","last_active":"2023-10-06T01:41:13.351608+0000","last_peered":"2023-10-06T01:41:13.351608+0000","last_clean":"2023-10-06T01:41:13.351608+0000","last_became_active":"2023-10-06T01:41:13.341868+0000","last_became_peered":"2023-10-06T01:41:13.341868+0000","last_unstale":"2023-10-06T01:41:13.351608+0000","last_undegraded":"2023-10-06T01:41:13.351608+0000","last_fullsized":"2023-10-06T01:41:13.351608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T01:51:57.229301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.329621+0000","last_change":"2023-10-06T01:41:12.339878+0000","last_active":"2023-10-06T01:41:13.329621+0000","last_peered":"2023-10-06T01:41:13.329621+0000","last_clean":"2023-10-06T01:41:13.329621+0000","last_became_active":"2023-10-06T01:41:12.338935+0000","last_became_peered":"2023-10-06T01:41:12.338935+0000","last_unstale":"2023-10-06T01:41:13.329621+0000","last_undegraded":"2023-10-06T01:41:13.329621+0000","last_fullsized":"2023-10-06T01:41:13.329621+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:44:53.014481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351640+0000","last_change":"2023-10-06T01:41:13.344292+0000","last_active":"2023-10-06T01:41:13.351640+0000","last_peered":"2023-10-06T01:41:13.351640+0000","last_clean":"2023-10-06T01:41:13.351640+0000","last_became_active":"2023-10-06T01:41:13.343947+0000","last_became_peered":"2023-10-06T01:41:13.343947+0000","last_unstale":"2023-10-06T01:41:13.351640+0000","last_undegraded":"2023-10-06T01:41:13.351640+0000","last_fullsized":"2023-10-06T01:41:13.351640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:45:07.571069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.331713+0000","last_change":"2023-10-06T01:41:12.336355+0000","last_active":"2023-10-06T01:41:13.331713+0000","last_peered":"2023-10-06T01:41:13.331713+0000","last_clean":"2023-10-06T01:41:13.331713+0000","last_became_active":"2023-10-06T01:41:12.336064+0000","last_became_peered":"2023-10-06T01:41:12.336064+0000","last_unstale":"2023-10-06T01:41:13.331713+0000","last_undegraded":"2023-10-06T01:41:13.331713+0000","last_fullsized":"2023-10-06T01:41:13.331713+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:16:14.826869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354641+0000","last_change":"2023-10-06T01:41:13.346404+0000","last_active":"2023-10-06T01:41:13.354641+0000","last_peered":"2023-10-06T01:41:13.354641+0000","last_clean":"2023-10-06T01:41:13.354641+0000","last_became_active":"2023-10-06T01:41:13.343805+0000","last_became_peered":"2023-10-06T01:41:13.343805+0000","last_unstale":"2023-10-06T01:41:13.354641+0000","last_undegraded":"2023-10-06T01:41:13.354641+0000","last_fullsized":"2023-10-06T01:41:13.354641+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:11:26.794025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-10-06T01:41:12.349596+0000","last_change":"2023-10-06T01:41:12.345661+0000","last_active":"2023-10-06T01:41:12.349596+0000","last_peered":"2023-10-06T01:41:12.349596+0000","last_clean":"2023-10-06T01:41:12.349596+0000","last_became_active":"2023-10-06T01:41:12.345363+0000","last_became_peered":"2023-10-06T01:41:12.345363+0000","last_unstale":"2023-10-06T01:41:12.349596+0000","last_undegraded":"2023-10-06T01:41:12.349596+0000","last_fullsized":"2023-10-06T01:41:12.349596+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:50:44.552768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356650+0000","last_change":"2023-10-06T01:41:13.353633+0000","last_active":"2023-10-06T01:41:13.356650+0000","last_peered":"2023-10-06T01:41:13.356650+0000","last_clean":"2023-10-06T01:41:13.356650+0000","last_became_active":"2023-10-06T01:41:13.353318+0000","last_became_peered":"2023-10-06T01:41:13.353318+0000","last_unstale":"2023-10-06T01:41:13.356650+0000","last_undegraded":"2023-10-06T01:41:13.356650+0000","last_fullsized":"2023-10-06T01:41:13.356650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:42:23.199959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"19'2","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354459+0000","last_change":"2023-10-06T01:41:12.340463+0000","last_active":"2023-10-06T01:41:13.354459+0000","last_peered":"2023-10-06T01:41:13.354459+0000","last_clean":"2023-10-06T01:41:13.354459+0000","last_became_active":"2023-10-06T01:41:12.340186+0000","last_became_peered":"2023-10-06T01:41:12.340186+0000","last_unstale":"2023-10-06T01:41:13.354459+0000","last_undegraded":"2023-10-06T01:41:13.354459+0000","last_fullsized":"2023-10-06T01:41:13.354459+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:16:14.826869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.330869+0000","last_change":"2023-10-06T01:41:07.454295+0000","last_active":"2023-10-06T01:41:13.330869+0000","last_peered":"2023-10-06T01:41:13.330869+0000","last_clean":"2023-10-06T01:41:13.330869+0000","last_became_active":"2023-10-06T01:41:07.454034+0000","last_became_peered":"2023-10-06T01:41:07.454034+0000","last_unstale":"2023-10-06T01:41:13.330869+0000","last_undegraded":"2023-10-06T01:41:13.330869+0000","last_fullsized":"2023-10-06T01:41:13.330869+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:32:58.914183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351289+0000","last_change":"2023-10-06T01:41:13.351289+0000","last_active":"2023-10-06T01:41:13.351289+0000","last_peered":"2023-10-06T01:41:13.351289+0000","last_clean":"2023-10-06T01:41:13.351289+0000","last_became_active":"2023-10-06T01:41:13.351022+0000","last_became_peered":"2023-10-06T01:41:13.351022+0000","last_unstale":"2023-10-06T01:41:13.351289+0000","last_undegraded":"2023-10-06T01:41:13.351289+0000","last_fullsized":"2023-10-06T01:41:13.351289+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T01:51:57.229301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":12,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-10-06T01:41:12.327374+0000","last_change":"2023-10-06T01:41:07.261955+0000","last_active":"2023-10-06T01:41:12.327374+0000","last_peered":"2023-10-06T01:41:12.327374+0000","last_clean":"2023-10-06T01:41:12.327374+0000","last_became_active":"2023-10-06T01:41:07.261692+0000","last_became_peered":"2023-10-06T01:41:07.261692+0000","last_unstale":"2023-10-06T01:41:12.327374+0000","last_undegraded":"2023-10-06T01:41:12.327374+0000","last_fullsized":"2023-10-06T01:41:12.327374+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-10-06T01:41:12.344691+0000","last_change":"2023-10-06T01:41:12.341312+0000","last_active":"2023-10-06T01:41:12.344691+0000","last_peered":"2023-10-06T01:41:12.344691+0000","last_clean":"2023-10-06T01:41:12.344691+0000","last_became_active":"2023-10-06T01:41:12.340997+0000","last_became_peered":"2023-10-06T01:41:12.340997+0000","last_unstale":"2023-10-06T01:41:12.344691+0000","last_undegraded":"2023-10-06T01:41:12.344691+0000","last_fullsized":"2023-10-06T01:41:12.344691+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:06:19.029137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351571+0000","last_change":"2023-10-06T01:41:13.344156+0000","last_active":"2023-10-06T01:41:13.351571+0000","last_peered":"2023-10-06T01:41:13.351571+0000","last_clean":"2023-10-06T01:41:13.351571+0000","last_became_active":"2023-10-06T01:41:13.343601+0000","last_became_peered":"2023-10-06T01:41:13.343601+0000","last_unstale":"2023-10-06T01:41:13.351571+0000","last_undegraded":"2023-10-06T01:41:13.351571+0000","last_fullsized":"2023-10-06T01:41:13.351571+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:21:42.851681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.336375+0000","last_change":"2023-10-06T01:41:07.258362+0000","last_active":"2023-10-06T01:41:13.336375+0000","last_peered":"2023-10-06T01:41:13.336375+0000","last_clean":"2023-10-06T01:41:13.336375+0000","last_became_active":"2023-10-06T01:41:07.258055+0000","last_became_peered":"2023-10-06T01:41:07.258055+0000","last_unstale":"2023-10-06T01:41:13.336375+0000","last_undegraded":"2023-10-06T01:41:13.336375+0000","last_fullsized":"2023-10-06T01:41:13.336375+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"16'117","reported_seq":163,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.334859+0000","last_change":"2023-10-06T01:41:07.261164+0000","last_active":"2023-10-06T01:41:13.334859+0000","last_peered":"2023-10-06T01:41:13.334859+0000","last_clean":"2023-10-06T01:41:13.334859+0000","last_became_active":"2023-10-06T01:41:07.260896+0000","last_became_peered":"2023-10-06T01:41:07.260896+0000","last_unstale":"2023-10-06T01:41:13.334859+0000","last_undegraded":"2023-10-06T01:41:13.334859+0000","last_fullsized":"2023-10-06T01:41:13.334859+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:05:22.934401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-10-06T01:41:12.348536+0000","last_change":"2023-10-06T01:41:12.343057+0000","last_active":"2023-10-06T01:41:12.348536+0000","last_peered":"2023-10-06T01:41:12.348536+0000","last_clean":"2023-10-06T01:41:12.348536+0000","last_became_active":"2023-10-06T01:41:12.342793+0000","last_became_peered":"2023-10-06T01:41:12.342793+0000","last_unstale":"2023-10-06T01:41:12.348536+0000","last_undegraded":"2023-10-06T01:41:12.348536+0000","last_fullsized":"2023-10-06T01:41:12.348536+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:15:40.841990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.346571+0000","last_change":"2023-10-06T01:41:13.346571+0000","last_active":"2023-10-06T01:41:13.346571+0000","last_peered":"2023-10-06T01:41:13.346571+0000","last_clean":"2023-10-06T01:41:13.346571+0000","last_became_active":"2023-10-06T01:41:13.346077+0000","last_became_peered":"2023-10-06T01:41:13.346077+0000","last_unstale":"2023-10-06T01:41:13.346571+0000","last_undegraded":"2023-10-06T01:41:13.346571+0000","last_fullsized":"2023-10-06T01:41:13.346571+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:42:51.581939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.333951+0000","last_change":"2023-10-06T01:41:07.456511+0000","last_active":"2023-10-06T01:41:13.333951+0000","last_peered":"2023-10-06T01:41:13.333951+0000","last_clean":"2023-10-06T01:41:13.333951+0000","last_became_active":"2023-10-06T01:41:07.456024+0000","last_became_peered":"2023-10-06T01:41:07.456024+0000","last_unstale":"2023-10-06T01:41:13.333951+0000","last_undegraded":"2023-10-06T01:41:13.333951+0000","last_fullsized":"2023-10-06T01:41:13.333951+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:51:02.949231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.336635+0000","last_change":"2023-10-06T01:41:12.343382+0000","last_active":"2023-10-06T01:41:13.336635+0000","last_peered":"2023-10-06T01:41:13.336635+0000","last_clean":"2023-10-06T01:41:13.336635+0000","last_became_active":"2023-10-06T01:41:12.343088+0000","last_became_peered":"2023-10-06T01:41:12.343088+0000","last_unstale":"2023-10-06T01:41:13.336635+0000","last_undegraded":"2023-10-06T01:41:13.336635+0000","last_fullsized":"2023-10-06T01:41:13.336635+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:44:53.014481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.350157+0000","last_change":"2023-10-06T01:41:13.346852+0000","last_active":"2023-10-06T01:41:13.350157+0000","last_peered":"2023-10-06T01:41:13.350157+0000","last_clean":"2023-10-06T01:41:13.350157+0000","last_became_active":"2023-10-06T01:41:13.346571+0000","last_became_peered":"2023-10-06T01:41:13.346571+0000","last_unstale":"2023-10-06T01:41:13.350157+0000","last_undegraded":"2023-10-06T01:41:13.350157+0000","last_fullsized":"2023-10-06T01:41:13.350157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:25:04.563235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.334453+0000","last_change":"2023-10-06T01:41:07.456273+0000","last_active":"2023-10-06T01:41:13.334453+0000","last_peered":"2023-10-06T01:41:13.334453+0000","last_clean":"2023-10-06T01:41:13.334453+0000","last_became_active":"2023-10-06T01:41:07.456000+0000","last_became_peered":"2023-10-06T01:41:07.456000+0000","last_unstale":"2023-10-06T01:41:13.334453+0000","last_undegraded":"2023-10-06T01:41:13.334453+0000","last_fullsized":"2023-10-06T01:41:13.334453+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:01:42.633388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-10-06T01:41:12.349428+0000","last_change":"2023-10-06T01:41:12.342215+0000","last_active":"2023-10-06T01:41:12.349428+0000","last_peered":"2023-10-06T01:41:12.349428+0000","last_clean":"2023-10-06T01:41:12.349428+0000","last_became_active":"2023-10-06T01:41:12.341493+0000","last_became_peered":"2023-10-06T01:41:12.341493+0000","last_unstale":"2023-10-06T01:41:12.349428+0000","last_undegraded":"2023-10-06T01:41:12.349428+0000","last_fullsized":"2023-10-06T01:41:12.349428+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:49:54.658088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352134+0000","last_change":"2023-10-06T01:41:13.346680+0000","last_active":"2023-10-06T01:41:13.352134+0000","last_peered":"2023-10-06T01:41:13.352134+0000","last_clean":"2023-10-06T01:41:13.352134+0000","last_became_active":"2023-10-06T01:41:13.346326+0000","last_became_peered":"2023-10-06T01:41:13.346326+0000","last_unstale":"2023-10-06T01:41:13.352134+0000","last_undegraded":"2023-10-06T01:41:13.352134+0000","last_fullsized":"2023-10-06T01:41:13.352134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:12:52.741339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.338955+0000","last_change":"2023-10-06T01:41:07.456688+0000","last_active":"2023-10-06T01:41:13.338955+0000","last_peered":"2023-10-06T01:41:13.338955+0000","last_clean":"2023-10-06T01:41:13.338955+0000","last_became_active":"2023-10-06T01:41:07.456304+0000","last_became_peered":"2023-10-06T01:41:07.456304+0000","last_unstale":"2023-10-06T01:41:13.338955+0000","last_undegraded":"2023-10-06T01:41:13.338955+0000","last_fullsized":"2023-10-06T01:41:13.338955+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.332203+0000","last_change":"2023-10-06T01:41:12.342354+0000","last_active":"2023-10-06T01:41:13.332203+0000","last_peered":"2023-10-06T01:41:13.332203+0000","last_clean":"2023-10-06T01:41:13.332203+0000","last_became_active":"2023-10-06T01:41:12.342066+0000","last_became_peered":"2023-10-06T01:41:12.342066+0000","last_unstale":"2023-10-06T01:41:13.332203+0000","last_undegraded":"2023-10-06T01:41:13.332203+0000","last_fullsized":"2023-10-06T01:41:13.332203+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:15:40.841990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352905+0000","last_change":"2023-10-06T01:41:13.348241+0000","last_active":"2023-10-06T01:41:13.352905+0000","last_peered":"2023-10-06T01:41:13.352905+0000","last_clean":"2023-10-06T01:41:13.352905+0000","last_became_active":"2023-10-06T01:41:13.347979+0000","last_became_peered":"2023-10-06T01:41:13.347979+0000","last_unstale":"2023-10-06T01:41:13.352905+0000","last_undegraded":"2023-10-06T01:41:13.352905+0000","last_fullsized":"2023-10-06T01:41:13.352905+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:41:59.186469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.329416+0000","last_change":"2023-10-06T01:41:07.456388+0000","last_active":"2023-10-06T01:41:13.329416+0000","last_peered":"2023-10-06T01:41:13.329416+0000","last_clean":"2023-10-06T01:41:13.329416+0000","last_became_active":"2023-10-06T01:41:07.456045+0000","last_became_peered":"2023-10-06T01:41:07.456045+0000","last_unstale":"2023-10-06T01:41:13.329416+0000","last_undegraded":"2023-10-06T01:41:13.329416+0000","last_fullsized":"2023-10-06T01:41:13.329416+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.338387+0000","last_change":"2023-10-06T01:41:12.344268+0000","last_active":"2023-10-06T01:41:13.338387+0000","last_peered":"2023-10-06T01:41:13.338387+0000","last_clean":"2023-10-06T01:41:13.338387+0000","last_became_active":"2023-10-06T01:41:12.343996+0000","last_became_peered":"2023-10-06T01:41:12.343996+0000","last_unstale":"2023-10-06T01:41:13.338387+0000","last_undegraded":"2023-10-06T01:41:13.338387+0000","last_fullsized":"2023-10-06T01:41:13.338387+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:37:52.143727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.346184+0000","last_change":"2023-10-06T01:41:13.346184+0000","last_active":"2023-10-06T01:41:13.346184+0000","last_peered":"2023-10-06T01:41:13.346184+0000","last_clean":"2023-10-06T01:41:13.346184+0000","last_became_active":"2023-10-06T01:41:13.345570+0000","last_became_peered":"2023-10-06T01:41:13.345570+0000","last_unstale":"2023-10-06T01:41:13.346184+0000","last_undegraded":"2023-10-06T01:41:13.346184+0000","last_fullsized":"2023-10-06T01:41:13.346184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:42:46.214805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.330470+0000","last_change":"2023-10-06T01:41:07.258162+0000","last_active":"2023-10-06T01:41:13.330470+0000","last_peered":"2023-10-06T01:41:13.330470+0000","last_clean":"2023-10-06T01:41:13.330470+0000","last_became_active":"2023-10-06T01:41:07.257795+0000","last_became_peered":"2023-10-06T01:41:07.257795+0000","last_unstale":"2023-10-06T01:41:13.330470+0000","last_undegraded":"2023-10-06T01:41:13.330470+0000","last_fullsized":"2023-10-06T01:41:13.330470+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:05:22.934401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.337002+0000","last_change":"2023-10-06T01:41:12.341990+0000","last_active":"2023-10-06T01:41:13.337002+0000","last_peered":"2023-10-06T01:41:13.337002+0000","last_clean":"2023-10-06T01:41:13.337002+0000","last_became_active":"2023-10-06T01:41:12.341680+0000","last_became_peered":"2023-10-06T01:41:12.341680+0000","last_unstale":"2023-10-06T01:41:13.337002+0000","last_undegraded":"2023-10-06T01:41:13.337002+0000","last_fullsized":"2023-10-06T01:41:13.337002+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:50:44.552768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.349056+0000","last_change":"2023-10-06T01:41:13.344589+0000","last_active":"2023-10-06T01:41:13.349056+0000","last_peered":"2023-10-06T01:41:13.349056+0000","last_clean":"2023-10-06T01:41:13.349056+0000","last_became_active":"2023-10-06T01:41:13.344281+0000","last_became_peered":"2023-10-06T01:41:13.344281+0000","last_unstale":"2023-10-06T01:41:13.349056+0000","last_undegraded":"2023-10-06T01:41:13.349056+0000","last_fullsized":"2023-10-06T01:41:13.349056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:32:17.938309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"19'1","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.329842+0000","last_change":"2023-10-06T01:41:12.347655+0000","last_active":"2023-10-06T01:41:13.329842+0000","last_peered":"2023-10-06T01:41:13.329842+0000","last_clean":"2023-10-06T01:41:13.329842+0000","last_became_active":"2023-10-06T01:41:12.347353+0000","last_became_peered":"2023-10-06T01:41:12.347353+0000","last_unstale":"2023-10-06T01:41:13.329842+0000","last_undegraded":"2023-10-06T01:41:13.329842+0000","last_fullsized":"2023-10-06T01:41:13.329842+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:49:54.658088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352446+0000","last_change":"2023-10-06T01:41:13.349874+0000","last_active":"2023-10-06T01:41:13.352446+0000","last_peered":"2023-10-06T01:41:13.352446+0000","last_clean":"2023-10-06T01:41:13.352446+0000","last_became_active":"2023-10-06T01:41:13.349607+0000","last_became_peered":"2023-10-06T01:41:13.349607+0000","last_unstale":"2023-10-06T01:41:13.352446+0000","last_undegraded":"2023-10-06T01:41:13.352446+0000","last_fullsized":"2023-10-06T01:41:13.352446+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:06:20.036081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352575+0000","last_change":"2023-10-06T01:41:12.341511+0000","last_active":"2023-10-06T01:41:13.352575+0000","last_peered":"2023-10-06T01:41:13.352575+0000","last_clean":"2023-10-06T01:41:13.352575+0000","last_became_active":"2023-10-06T01:41:12.341216+0000","last_became_peered":"2023-10-06T01:41:12.341216+0000","last_unstale":"2023-10-06T01:41:13.352575+0000","last_undegraded":"2023-10-06T01:41:13.352575+0000","last_fullsized":"2023-10-06T01:41:13.352575+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:27:01.582307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351635+0000","last_change":"2023-10-06T01:41:13.343780+0000","last_active":"2023-10-06T01:41:13.351635+0000","last_peered":"2023-10-06T01:41:13.351635+0000","last_clean":"2023-10-06T01:41:13.351635+0000","last_became_active":"2023-10-06T01:41:13.343515+0000","last_became_peered":"2023-10-06T01:41:13.343515+0000","last_unstale":"2023-10-06T01:41:13.351635+0000","last_undegraded":"2023-10-06T01:41:13.351635+0000","last_fullsized":"2023-10-06T01:41:13.351635+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:38:58.407819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.353702+0000","last_change":"2023-10-06T01:41:12.341780+0000","last_active":"2023-10-06T01:41:13.353702+0000","last_peered":"2023-10-06T01:41:13.353702+0000","last_clean":"2023-10-06T01:41:13.353702+0000","last_became_active":"2023-10-06T01:41:12.340585+0000","last_became_peered":"2023-10-06T01:41:12.340585+0000","last_unstale":"2023-10-06T01:41:13.353702+0000","last_undegraded":"2023-10-06T01:41:13.353702+0000","last_fullsized":"2023-10-06T01:41:13.353702+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:49:54.658088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354558+0000","last_change":"2023-10-06T01:41:13.345543+0000","last_active":"2023-10-06T01:41:13.354558+0000","last_peered":"2023-10-06T01:41:13.354558+0000","last_clean":"2023-10-06T01:41:13.354558+0000","last_became_active":"2023-10-06T01:41:13.345286+0000","last_became_peered":"2023-10-06T01:41:13.345286+0000","last_unstale":"2023-10-06T01:41:13.354558+0000","last_undegraded":"2023-10-06T01:41:13.354558+0000","last_fullsized":"2023-10-06T01:41:13.354558+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:52:48.286318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"19'1","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.337057+0000","last_change":"2023-10-06T01:41:12.340776+0000","last_active":"2023-10-06T01:41:13.337057+0000","last_peered":"2023-10-06T01:41:13.337057+0000","last_clean":"2023-10-06T01:41:13.337057+0000","last_became_active":"2023-10-06T01:41:12.340294+0000","last_became_peered":"2023-10-06T01:41:12.340294+0000","last_unstale":"2023-10-06T01:41:13.337057+0000","last_undegraded":"2023-10-06T01:41:13.337057+0000","last_fullsized":"2023-10-06T01:41:13.337057+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:37:13.672716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.355273+0000","last_change":"2023-10-06T01:41:13.355273+0000","last_active":"2023-10-06T01:41:13.355273+0000","last_peered":"2023-10-06T01:41:13.355273+0000","last_clean":"2023-10-06T01:41:13.355273+0000","last_became_active":"2023-10-06T01:41:13.354977+0000","last_became_peered":"2023-10-06T01:41:13.354977+0000","last_unstale":"2023-10-06T01:41:13.355273+0000","last_undegraded":"2023-10-06T01:41:13.355273+0000","last_fullsized":"2023-10-06T01:41:13.355273+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:21:42.851681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"19'4","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.345488+0000","last_change":"2023-10-06T01:41:12.338431+0000","last_active":"2023-10-06T01:41:13.345488+0000","last_peered":"2023-10-06T01:41:13.345488+0000","last_clean":"2023-10-06T01:41:13.345488+0000","last_became_active":"2023-10-06T01:41:12.338081+0000","last_became_peered":"2023-10-06T01:41:12.338081+0000","last_unstale":"2023-10-06T01:41:13.345488+0000","last_undegraded":"2023-10-06T01:41:13.345488+0000","last_fullsized":"2023-10-06T01:41:13.345488+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:16:14.826869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351157+0000","last_change":"2023-10-06T01:41:13.342212+0000","last_active":"2023-10-06T01:41:13.351157+0000","last_peered":"2023-10-06T01:41:13.351157+0000","last_clean":"2023-10-06T01:41:13.351157+0000","last_became_active":"2023-10-06T01:41:13.341948+0000","last_became_peered":"2023-10-06T01:41:13.341948+0000","last_unstale":"2023-10-06T01:41:13.351157+0000","last_undegraded":"2023-10-06T01:41:13.351157+0000","last_fullsized":"2023-10-06T01:41:13.351157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:18:23.688664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354188+0000","last_change":"2023-10-06T01:41:13.348482+0000","last_active":"2023-10-06T01:41:13.354188+0000","last_peered":"2023-10-06T01:41:13.354188+0000","last_clean":"2023-10-06T01:41:13.354188+0000","last_became_active":"2023-10-06T01:41:13.348075+0000","last_became_peered":"2023-10-06T01:41:13.348075+0000","last_unstale":"2023-10-06T01:41:13.354188+0000","last_undegraded":"2023-10-06T01:41:13.354188+0000","last_fullsized":"2023-10-06T01:41:13.354188+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:25:04.563235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.355216+0000","last_change":"2023-10-06T01:41:13.355216+0000","last_active":"2023-10-06T01:41:13.355216+0000","last_peered":"2023-10-06T01:41:13.355216+0000","last_clean":"2023-10-06T01:41:13.355216+0000","last_became_active":"2023-10-06T01:41:13.354853+0000","last_became_peered":"2023-10-06T01:41:13.354853+0000","last_unstale":"2023-10-06T01:41:13.355216+0000","last_undegraded":"2023-10-06T01:41:13.355216+0000","last_fullsized":"2023-10-06T01:41:13.355216+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:42:44.799267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351163+0000","last_change":"2023-10-06T01:41:13.342888+0000","last_active":"2023-10-06T01:41:13.351163+0000","last_peered":"2023-10-06T01:41:13.351163+0000","last_clean":"2023-10-06T01:41:13.351163+0000","last_became_active":"2023-10-06T01:41:13.342626+0000","last_became_peered":"2023-10-06T01:41:13.342626+0000","last_unstale":"2023-10-06T01:41:13.351163+0000","last_undegraded":"2023-10-06T01:41:13.351163+0000","last_fullsized":"2023-10-06T01:41:13.351163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:43:29.491858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":8,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.353822+0000","last_change":"2023-10-06T01:41:13.353822+0000","last_active":"2023-10-06T01:41:13.353822+0000","last_peered":"2023-10-06T01:41:13.353822+0000","last_clean":"2023-10-06T01:41:13.353822+0000","last_became_active":"2023-10-06T01:41:13.353292+0000","last_became_peered":"2023-10-06T01:41:13.353292+0000","last_unstale":"2023-10-06T01:41:13.353822+0000","last_undegraded":"2023-10-06T01:41:13.353822+0000","last_fullsized":"2023-10-06T01:41:13.353822+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:52:48.286318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356636+0000","last_change":"2023-10-06T01:41:13.353662+0000","last_active":"2023-10-06T01:41:13.356636+0000","last_peered":"2023-10-06T01:41:13.356636+0000","last_clean":"2023-10-06T01:41:13.356636+0000","last_became_active":"2023-10-06T01:41:13.353269+0000","last_became_peered":"2023-10-06T01:41:13.353269+0000","last_unstale":"2023-10-06T01:41:13.356636+0000","last_undegraded":"2023-10-06T01:41:13.356636+0000","last_fullsized":"2023-10-06T01:41:13.356636+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:11:44.715827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298096,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073744,"statfs":{"total":96636764160,"available":96331513856,"internally_reserved":0,"allocated":827392,"data_stored":666304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38200000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72199999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69999999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297484,"kb_used_data":196,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074356,"statfs":{"total":96636764160,"available":96332140544,"internally_reserved":0,"allocated":200704,"data_stored":72831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48699999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298068,"kb_used_data":780,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073772,"statfs":{"total":96636764160,"available":96331542528,"internally_reserved":0,"allocated":798720,"data_stored":664552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34100000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56799999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297476,"kb_used_data":188,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074364,"statfs":{"total":96636764160,"available":96332148736,"internally_reserved":0,"allocated":192512,"data_stored":66744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57099999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65500000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72899999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297488,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074352,"statfs":{"total":96636764160,"available":96332136448,"internally_reserved":0,"allocated":204800,"data_stored":74184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61099999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297476,"kb_used_data":188,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074364,"statfs":{"total":96636764160,"available":96332148736,"internally_reserved":0,"allocated":192512,"data_stored":66280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297508,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074332,"statfs":{"total":96636764160,"available":96332115968,"internally_reserved":0,"allocated":225280,"data_stored":75776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297492,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074348,"statfs":{"total":96636764160,"available":96332132352,"internally_reserved":0,"allocated":208896,"data_stored":73455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57199999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-06T01:41:22.015 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-06T01:41:22.148 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.147+0000 7f8767c89700 1 Processor -- start 2023-10-06T01:41:22.148 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.148+0000 7f8767c89700 1 -- start start 2023-10-06T01:41:22.150 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.148+0000 7f8767c89700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f87600add70 0x7f87600ae190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:22.150 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.148+0000 7f8767c89700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87601580d0 con 0x7f87600b4060 2023-10-06T01:41:22.150 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.148+0000 7f8767c89700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87600ae6d0 con 0x7f8760157570 2023-10-06T01:41:22.150 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.148+0000 7f8767c89700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8760158c70 con 0x7f87600add70 2023-10-06T01:41:22.150 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.149+0000 7f8765224700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f87600b4060 0x7f8760158560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46224/0 (socket says 172.21.15.27:46224) 2023-10-06T01:41:22.150 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.149+0000 7f8765224700 1 -- 172.21.15.27:0/1329567009 learned_addr learned my addr 172.21.15.27:0/1329567009 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:22.150 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.149+0000 7f8766226700 1 --2- 172.21.15.27:0/1329567009 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f87600add70 0x7f87600ae190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:22.150 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.149+0000 7f8764a23700 1 -- 172.21.15.27:0/1329567009 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2456550087 0 0) 0x7f87601580d0 con 0x7f87600b4060 2023-10-06T01:41:22.150 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.149+0000 7f8764a23700 1 -- 172.21.15.27:0/1329567009 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8748003570 con 0x7f87600b4060 2023-10-06T01:41:22.150 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.149+0000 7f8764a23700 1 -- 172.21.15.27:0/1329567009 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4288395557 0 0) 0x7f87600ae6d0 con 0x7f8760157570 2023-10-06T01:41:22.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.149+0000 7f8764a23700 1 -- 172.21.15.27:0/1329567009 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87601580d0 con 0x7f8760157570 2023-10-06T01:41:22.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.149+0000 7f8766226700 1 -- 172.21.15.27:0/1329567009 >> v1:172.21.15.124:6789/0 conn(0x7f8760157570 legacy=0x7f876015b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:22.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.149+0000 7f8766226700 1 -- 172.21.15.27:0/1329567009 >> v1:172.21.15.27:6789/0 conn(0x7f87600b4060 legacy=0x7f8760158560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:22.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.149+0000 7f8766226700 1 -- 172.21.15.27:0/1329567009 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f876015ba40 con 0x7f87600add70 2023-10-06T01:41:22.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.149+0000 7f8766226700 1 --2- 172.21.15.27:0/1329567009 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f87600add70 0x7f87600ae190 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8758009910 tx=0x7f87580091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e72d967127435e6 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:22.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.149+0000 7f8764a23700 1 -- 172.21.15.27:0/1329567009 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8758011630 con 0x7f87600add70 2023-10-06T01:41:22.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.149+0000 7f8764a23700 1 -- 172.21.15.27:0/1329567009 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8758011790 con 0x7f87600add70 2023-10-06T01:41:22.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.149+0000 7f8764a23700 1 -- 172.21.15.27:0/1329567009 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8758011b30 con 0x7f87600add70 2023-10-06T01:41:22.154 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.150+0000 7f8767c89700 1 -- 172.21.15.27:0/1329567009 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f87600add70 msgr2=0x7f87600ae190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:22.154 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.150+0000 7f8767c89700 1 --2- 172.21.15.27:0/1329567009 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f87600add70 0x7f87600ae190 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f8758009910 tx=0x7f87580091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:22.154 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.150+0000 7f8767c89700 1 -- 172.21.15.27:0/1329567009 shutdown_connections 2023-10-06T01:41:22.154 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.150+0000 7f8767c89700 1 --2- 172.21.15.27:0/1329567009 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f87600add70 0x7f87600ae190 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:22.154 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.150+0000 7f8767c89700 1 -- 172.21.15.27:0/1329567009 >> 172.21.15.27:0/1329567009 conn(0x7f8760067630 msgr2=0x7f8760067a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:22.154 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.150+0000 7f8767c89700 1 -- 172.21.15.27:0/1329567009 shutdown_connections 2023-10-06T01:41:22.154 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.150+0000 7f8767c89700 1 -- 172.21.15.27:0/1329567009 wait complete. 2023-10-06T01:41:22.154 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.151+0000 7f8767c89700 1 Processor -- start 2023-10-06T01:41:22.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.151+0000 7f8767c89700 1 -- start start 2023-10-06T01:41:22.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.152+0000 7f8767c89700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87600add70 0x7f87600c49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:22.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.152+0000 7f8767c89700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f87600b4060 0x7f87600c9220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:22.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.152+0000 7f8767c89700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8760157570 0x7f87600c9760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:22.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.152+0000 7f8767c89700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87600c9dc0 con 0x7f87600add70 2023-10-06T01:41:22.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.152+0000 7f8767c89700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87600ca0a0 con 0x7f8760157570 2023-10-06T01:41:22.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.152+0000 7f8767c89700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87600ca380 con 0x7f87600b4060 2023-10-06T01:41:22.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.152+0000 7f8765a25700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87600add70 0x7f87600c49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:22.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.152+0000 7f8765a25700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87600add70 0x7f87600c49d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55826/0 (socket says 172.21.15.27:55826) 2023-10-06T01:41:22.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.152+0000 7f8765a25700 1 -- 172.21.15.27:0/3515123937 learned_addr learned my addr 172.21.15.27:0/3515123937 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:22.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.152+0000 7f8766226700 1 --2- 172.21.15.27:0/3515123937 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8760157570 0x7f87600c9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:22.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.152+0000 7f8765224700 1 --2- 172.21.15.27:0/3515123937 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f87600b4060 0x7f87600c9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:22.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.152+0000 7f8765a25700 1 -- 172.21.15.27:0/3515123937 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f87600b4060 msgr2=0x7f87600c9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:22.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.152+0000 7f8765a25700 1 --2- 172.21.15.27:0/3515123937 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f87600b4060 0x7f87600c9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:22.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.152+0000 7f8765a25700 1 -- 172.21.15.27:0/3515123937 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8760157570 msgr2=0x7f87600c9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:22.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.152+0000 7f8765a25700 1 --2- 172.21.15.27:0/3515123937 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8760157570 0x7f87600c9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:22.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.152+0000 7f8765a25700 1 -- 172.21.15.27:0/3515123937 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8758009420 con 0x7f87600add70 2023-10-06T01:41:22.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.152+0000 7f8766226700 1 --2- 172.21.15.27:0/3515123937 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8760157570 0x7f87600c9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:22.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.152+0000 7f8765224700 1 --2- 172.21.15.27:0/3515123937 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f87600b4060 0x7f87600c9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:22.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.153+0000 7f8765a25700 1 --2- 172.21.15.27:0/3515123937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87600add70 0x7f87600c49d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f875000cb40 tx=0x7f8750002810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:22.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.153+0000 7f875dffb700 1 -- 172.21.15.27:0/3515123937 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875000a7e0 con 0x7f87600add70 2023-10-06T01:41:22.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.153+0000 7f875dffb700 1 -- 172.21.15.27:0/3515123937 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f875000a960 con 0x7f87600add70 2023-10-06T01:41:22.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.153+0000 7f8767c89700 1 -- 172.21.15.27:0/3515123937 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87600c54b0 con 0x7f87600add70 2023-10-06T01:41:22.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.153+0000 7f875dffb700 1 -- 172.21.15.27:0/3515123937 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875000ad20 con 0x7f87600add70 2023-10-06T01:41:22.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.153+0000 7f8767c89700 1 -- 172.21.15.27:0/3515123937 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87600acfe0 con 0x7f87600add70 2023-10-06T01:41:22.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.154+0000 7f8767c89700 1 -- 172.21.15.27:0/3515123937 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8760064050 con 0x7f87600add70 2023-10-06T01:41:22.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.154+0000 7f875dffb700 1 -- 172.21.15.27:0/3515123937 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8750012630 con 0x7f87600add70 2023-10-06T01:41:22.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.155+0000 7f875dffb700 1 --2- 172.21.15.27:0/3515123937 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f874c066f30 0x7f874c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:22.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.155+0000 7f875dffb700 1 -- 172.21.15.27:0/3515123937 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f8750084a60 con 0x7f87600add70 2023-10-06T01:41:22.158 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.155+0000 7f8765224700 1 --2- 172.21.15.27:0/3515123937 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f874c066f30 0x7f874c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:22.158 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.155+0000 7f8765224700 1 --2- 172.21.15.27:0/3515123937 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f874c066f30 0x7f874c0693e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f87540058d0 tx=0x7f8754009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:22.159 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.159+0000 7f875dffb700 1 -- 172.21.15.27:0/3515123937 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8750051b70 con 0x7f87600add70 2023-10-06T01:41:22.348 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.348+0000 7f8767c89700 1 -- 172.21.15.27:0/3515123937 --> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f87600cdf50 con 0x7f874c066f30 2023-10-06T01:41:22.357 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.356+0000 7f875dffb700 1 -- 172.21.15.27:0/3515123937 <== mgr.4100 v2:172.21.15.124:6832/90278 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286788 (secure 0 0 0) 0x7f87600cdf50 con 0x7f874c066f30 2023-10-06T01:41:22.360 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:22.362 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.359+0000 7f8767c89700 1 -- 172.21.15.27:0/3515123937 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f874c066f30 msgr2=0x7f874c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:22.362 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.359+0000 7f8767c89700 1 --2- 172.21.15.27:0/3515123937 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f874c066f30 0x7f874c0693e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f87540058d0 tx=0x7f8754009040 comp rx=0 tx=0).stop 2023-10-06T01:41:22.363 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.359+0000 7f8767c89700 1 -- 172.21.15.27:0/3515123937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87600add70 msgr2=0x7f87600c49d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:22.363 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.359+0000 7f8767c89700 1 --2- 172.21.15.27:0/3515123937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87600add70 0x7f87600c49d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f875000cb40 tx=0x7f8750002810 comp rx=0 tx=0).stop 2023-10-06T01:41:22.363 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.359+0000 7f8767c89700 1 -- 172.21.15.27:0/3515123937 shutdown_connections 2023-10-06T01:41:22.363 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.359+0000 7f8767c89700 1 --2- 172.21.15.27:0/3515123937 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f874c066f30 0x7f874c0693e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:22.363 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.359+0000 7f8767c89700 1 --2- 172.21.15.27:0/3515123937 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8760157570 0x7f87600c9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:22.363 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.359+0000 7f8767c89700 1 --2- 172.21.15.27:0/3515123937 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f87600add70 0x7f87600c49d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:22.363 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.359+0000 7f8767c89700 1 --2- 172.21.15.27:0/3515123937 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f87600b4060 0x7f87600c9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:22.363 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.360+0000 7f8767c89700 1 -- 172.21.15.27:0/3515123937 >> 172.21.15.27:0/3515123937 conn(0x7f8760067630 msgr2=0x7f8760156b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:22.363 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.360+0000 7f8767c89700 1 -- 172.21.15.27:0/3515123937 shutdown_connections 2023-10-06T01:41:22.363 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.360+0000 7f8767c89700 1 -- 172.21.15.27:0/3515123937 wait complete. 2023-10-06T01:41:22.363 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-06T01:41:22.372 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2023-10-06T01:41:22.039988+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2381112,"kb_used_data":2808,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752593608,"statfs":{"total":773094113280,"available":770655854592,"internally_reserved":0,"allocated":2875392,"data_stored":1773408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.712280"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.349688+0000","last_change":"2023-10-06T01:41:13.346440+0000","last_active":"2023-10-06T01:41:13.349688+0000","last_peered":"2023-10-06T01:41:13.349688+0000","last_clean":"2023-10-06T01:41:13.349688+0000","last_became_active":"2023-10-06T01:41:13.343345+0000","last_became_peered":"2023-10-06T01:41:13.343345+0000","last_unstale":"2023-10-06T01:41:13.349688+0000","last_undegraded":"2023-10-06T01:41:13.349688+0000","last_fullsized":"2023-10-06T01:41:13.349688+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:15:41.848935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354765+0000","last_change":"2023-10-06T01:41:13.348847+0000","last_active":"2023-10-06T01:41:13.354765+0000","last_peered":"2023-10-06T01:41:13.354765+0000","last_clean":"2023-10-06T01:41:13.354765+0000","last_became_active":"2023-10-06T01:41:13.348585+0000","last_became_peered":"2023-10-06T01:41:13.348585+0000","last_unstale":"2023-10-06T01:41:13.354765+0000","last_undegraded":"2023-10-06T01:41:13.354765+0000","last_fullsized":"2023-10-06T01:41:13.354765+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:31:20.840764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354463+0000","last_change":"2023-10-06T01:41:13.345122+0000","last_active":"2023-10-06T01:41:13.354463+0000","last_peered":"2023-10-06T01:41:13.354463+0000","last_clean":"2023-10-06T01:41:13.354463+0000","last_became_active":"2023-10-06T01:41:13.344752+0000","last_became_peered":"2023-10-06T01:41:13.344752+0000","last_unstale":"2023-10-06T01:41:13.354463+0000","last_undegraded":"2023-10-06T01:41:13.354463+0000","last_fullsized":"2023-10-06T01:41:13.354463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:42:56.041827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.342734+0000","last_change":"2023-10-06T01:41:13.340589+0000","last_active":"2023-10-06T01:41:13.342734+0000","last_peered":"2023-10-06T01:41:13.342734+0000","last_clean":"2023-10-06T01:41:13.342734+0000","last_became_active":"2023-10-06T01:41:13.340339+0000","last_became_peered":"2023-10-06T01:41:13.340339+0000","last_unstale":"2023-10-06T01:41:13.342734+0000","last_undegraded":"2023-10-06T01:41:13.342734+0000","last_fullsized":"2023-10-06T01:41:13.342734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:32:17.938309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.346061+0000","last_change":"2023-10-06T01:41:13.342002+0000","last_active":"2023-10-06T01:41:13.346061+0000","last_peered":"2023-10-06T01:41:13.346061+0000","last_clean":"2023-10-06T01:41:13.346061+0000","last_became_active":"2023-10-06T01:41:13.341403+0000","last_became_peered":"2023-10-06T01:41:13.341403+0000","last_unstale":"2023-10-06T01:41:13.346061+0000","last_undegraded":"2023-10-06T01:41:13.346061+0000","last_fullsized":"2023-10-06T01:41:13.346061+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:34:55.375989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354785+0000","last_change":"2023-10-06T01:41:13.345138+0000","last_active":"2023-10-06T01:41:13.354785+0000","last_peered":"2023-10-06T01:41:13.354785+0000","last_clean":"2023-10-06T01:41:13.354785+0000","last_became_active":"2023-10-06T01:41:13.344810+0000","last_became_peered":"2023-10-06T01:41:13.344810+0000","last_unstale":"2023-10-06T01:41:13.354785+0000","last_undegraded":"2023-10-06T01:41:13.354785+0000","last_fullsized":"2023-10-06T01:41:13.354785+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:17:59.027125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356675+0000","last_change":"2023-10-06T01:41:13.352141+0000","last_active":"2023-10-06T01:41:13.356675+0000","last_peered":"2023-10-06T01:41:13.356675+0000","last_clean":"2023-10-06T01:41:13.356675+0000","last_became_active":"2023-10-06T01:41:13.351628+0000","last_became_peered":"2023-10-06T01:41:13.351628+0000","last_unstale":"2023-10-06T01:41:13.356675+0000","last_undegraded":"2023-10-06T01:41:13.356675+0000","last_fullsized":"2023-10-06T01:41:13.356675+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:03:37.385158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354278+0000","last_change":"2023-10-06T01:41:13.345960+0000","last_active":"2023-10-06T01:41:13.354278+0000","last_peered":"2023-10-06T01:41:13.354278+0000","last_clean":"2023-10-06T01:41:13.354278+0000","last_became_active":"2023-10-06T01:41:13.345702+0000","last_became_peered":"2023-10-06T01:41:13.345702+0000","last_unstale":"2023-10-06T01:41:13.354278+0000","last_undegraded":"2023-10-06T01:41:13.354278+0000","last_fullsized":"2023-10-06T01:41:13.354278+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:03:37.385158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.348168+0000","last_change":"2023-10-06T01:41:13.343792+0000","last_active":"2023-10-06T01:41:13.348168+0000","last_peered":"2023-10-06T01:41:13.348168+0000","last_clean":"2023-10-06T01:41:13.348168+0000","last_became_active":"2023-10-06T01:41:13.343432+0000","last_became_peered":"2023-10-06T01:41:13.343432+0000","last_unstale":"2023-10-06T01:41:13.348168+0000","last_undegraded":"2023-10-06T01:41:13.348168+0000","last_fullsized":"2023-10-06T01:41:13.348168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:52:48.286318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354747+0000","last_change":"2023-10-06T01:41:13.347399+0000","last_active":"2023-10-06T01:41:13.354747+0000","last_peered":"2023-10-06T01:41:13.354747+0000","last_clean":"2023-10-06T01:41:13.354747+0000","last_became_active":"2023-10-06T01:41:13.347136+0000","last_became_peered":"2023-10-06T01:41:13.347136+0000","last_unstale":"2023-10-06T01:41:13.354747+0000","last_undegraded":"2023-10-06T01:41:13.354747+0000","last_fullsized":"2023-10-06T01:41:13.354747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:42:23.199959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354772+0000","last_change":"2023-10-06T01:41:13.350059+0000","last_active":"2023-10-06T01:41:13.354772+0000","last_peered":"2023-10-06T01:41:13.354772+0000","last_clean":"2023-10-06T01:41:13.354772+0000","last_became_active":"2023-10-06T01:41:13.349673+0000","last_became_peered":"2023-10-06T01:41:13.349673+0000","last_unstale":"2023-10-06T01:41:13.354772+0000","last_undegraded":"2023-10-06T01:41:13.354772+0000","last_fullsized":"2023-10-06T01:41:13.354772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:12:57.315612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351538+0000","last_change":"2023-10-06T01:41:13.348001+0000","last_active":"2023-10-06T01:41:13.351538+0000","last_peered":"2023-10-06T01:41:13.351538+0000","last_clean":"2023-10-06T01:41:13.351538+0000","last_became_active":"2023-10-06T01:41:13.347719+0000","last_became_peered":"2023-10-06T01:41:13.347719+0000","last_unstale":"2023-10-06T01:41:13.351538+0000","last_undegraded":"2023-10-06T01:41:13.351538+0000","last_fullsized":"2023-10-06T01:41:13.351538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:37:53.150671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.355181+0000","last_change":"2023-10-06T01:41:13.352875+0000","last_active":"2023-10-06T01:41:13.355181+0000","last_peered":"2023-10-06T01:41:13.355181+0000","last_clean":"2023-10-06T01:41:13.355181+0000","last_became_active":"2023-10-06T01:41:13.352589+0000","last_became_peered":"2023-10-06T01:41:13.352589+0000","last_unstale":"2023-10-06T01:41:13.355181+0000","last_undegraded":"2023-10-06T01:41:13.355181+0000","last_fullsized":"2023-10-06T01:41:13.355181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:49:55.665032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354387+0000","last_change":"2023-10-06T01:41:13.348362+0000","last_active":"2023-10-06T01:41:13.354387+0000","last_peered":"2023-10-06T01:41:13.354387+0000","last_clean":"2023-10-06T01:41:13.354387+0000","last_became_active":"2023-10-06T01:41:13.348181+0000","last_became_peered":"2023-10-06T01:41:13.348181+0000","last_unstale":"2023-10-06T01:41:13.354387+0000","last_undegraded":"2023-10-06T01:41:13.354387+0000","last_fullsized":"2023-10-06T01:41:13.354387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:27:54.074728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356697+0000","last_change":"2023-10-06T01:41:13.352048+0000","last_active":"2023-10-06T01:41:13.356697+0000","last_peered":"2023-10-06T01:41:13.356697+0000","last_clean":"2023-10-06T01:41:13.356697+0000","last_became_active":"2023-10-06T01:41:13.351640+0000","last_became_peered":"2023-10-06T01:41:13.351640+0000","last_unstale":"2023-10-06T01:41:13.356697+0000","last_undegraded":"2023-10-06T01:41:13.356697+0000","last_fullsized":"2023-10-06T01:41:13.356697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:42:23.199959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354602+0000","last_change":"2023-10-06T01:41:13.345677+0000","last_active":"2023-10-06T01:41:13.354602+0000","last_peered":"2023-10-06T01:41:13.354602+0000","last_clean":"2023-10-06T01:41:13.354602+0000","last_became_active":"2023-10-06T01:41:13.345257+0000","last_became_peered":"2023-10-06T01:41:13.345257+0000","last_unstale":"2023-10-06T01:41:13.354602+0000","last_undegraded":"2023-10-06T01:41:13.354602+0000","last_fullsized":"2023-10-06T01:41:13.354602+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:27:07.040029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.348233+0000","last_change":"2023-10-06T01:41:13.344490+0000","last_active":"2023-10-06T01:41:13.348233+0000","last_peered":"2023-10-06T01:41:13.348233+0000","last_clean":"2023-10-06T01:41:13.348233+0000","last_became_active":"2023-10-06T01:41:13.344220+0000","last_became_peered":"2023-10-06T01:41:13.344220+0000","last_unstale":"2023-10-06T01:41:13.348233+0000","last_undegraded":"2023-10-06T01:41:13.348233+0000","last_fullsized":"2023-10-06T01:41:13.348233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T01:51:57.229301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354756+0000","last_change":"2023-10-06T01:41:13.350575+0000","last_active":"2023-10-06T01:41:13.354756+0000","last_peered":"2023-10-06T01:41:13.354756+0000","last_clean":"2023-10-06T01:41:13.354756+0000","last_became_active":"2023-10-06T01:41:13.350311+0000","last_became_peered":"2023-10-06T01:41:13.350311+0000","last_unstale":"2023-10-06T01:41:13.354756+0000","last_undegraded":"2023-10-06T01:41:13.354756+0000","last_fullsized":"2023-10-06T01:41:13.354756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:39:03.091509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351594+0000","last_change":"2023-10-06T01:41:13.347386+0000","last_active":"2023-10-06T01:41:13.351594+0000","last_peered":"2023-10-06T01:41:13.351594+0000","last_clean":"2023-10-06T01:41:13.351594+0000","last_became_active":"2023-10-06T01:41:13.347101+0000","last_became_peered":"2023-10-06T01:41:13.347101+0000","last_unstale":"2023-10-06T01:41:13.351594+0000","last_undegraded":"2023-10-06T01:41:13.351594+0000","last_fullsized":"2023-10-06T01:41:13.351594+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:13:15.011521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351504+0000","last_change":"2023-10-06T01:41:13.342640+0000","last_active":"2023-10-06T01:41:13.351504+0000","last_peered":"2023-10-06T01:41:13.351504+0000","last_clean":"2023-10-06T01:41:13.351504+0000","last_became_active":"2023-10-06T01:41:13.342157+0000","last_became_peered":"2023-10-06T01:41:13.342157+0000","last_unstale":"2023-10-06T01:41:13.351504+0000","last_undegraded":"2023-10-06T01:41:13.351504+0000","last_fullsized":"2023-10-06T01:41:13.351504+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:07:52.521701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356729+0000","last_change":"2023-10-06T01:41:13.352582+0000","last_active":"2023-10-06T01:41:13.356729+0000","last_peered":"2023-10-06T01:41:13.356729+0000","last_clean":"2023-10-06T01:41:13.356729+0000","last_became_active":"2023-10-06T01:41:13.352281+0000","last_became_peered":"2023-10-06T01:41:13.352281+0000","last_unstale":"2023-10-06T01:41:13.356729+0000","last_undegraded":"2023-10-06T01:41:13.356729+0000","last_fullsized":"2023-10-06T01:41:13.356729+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:31:20.840764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351592+0000","last_change":"2023-10-06T01:41:13.345324+0000","last_active":"2023-10-06T01:41:13.351592+0000","last_peered":"2023-10-06T01:41:13.351592+0000","last_clean":"2023-10-06T01:41:13.351592+0000","last_became_active":"2023-10-06T01:41:13.344997+0000","last_became_peered":"2023-10-06T01:41:13.344997+0000","last_unstale":"2023-10-06T01:41:13.351592+0000","last_undegraded":"2023-10-06T01:41:13.351592+0000","last_fullsized":"2023-10-06T01:41:13.351592+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:27:54.074728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.357943+0000","last_change":"2023-10-06T01:41:13.355108+0000","last_active":"2023-10-06T01:41:13.357943+0000","last_peered":"2023-10-06T01:41:13.357943+0000","last_clean":"2023-10-06T01:41:13.357943+0000","last_became_active":"2023-10-06T01:41:13.354621+0000","last_became_peered":"2023-10-06T01:41:13.354621+0000","last_unstale":"2023-10-06T01:41:13.357943+0000","last_undegraded":"2023-10-06T01:41:13.357943+0000","last_fullsized":"2023-10-06T01:41:13.357943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:34:55.375989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356702+0000","last_change":"2023-10-06T01:41:13.352301+0000","last_active":"2023-10-06T01:41:13.356702+0000","last_peered":"2023-10-06T01:41:13.356702+0000","last_clean":"2023-10-06T01:41:13.356702+0000","last_became_active":"2023-10-06T01:41:13.351858+0000","last_became_peered":"2023-10-06T01:41:13.351858+0000","last_unstale":"2023-10-06T01:41:13.356702+0000","last_undegraded":"2023-10-06T01:41:13.356702+0000","last_fullsized":"2023-10-06T01:41:13.356702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:42:46.214805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.348956+0000","last_change":"2023-10-06T01:41:13.344094+0000","last_active":"2023-10-06T01:41:13.348956+0000","last_peered":"2023-10-06T01:41:13.348956+0000","last_clean":"2023-10-06T01:41:13.348956+0000","last_became_active":"2023-10-06T01:41:13.343796+0000","last_became_peered":"2023-10-06T01:41:13.343796+0000","last_unstale":"2023-10-06T01:41:13.348956+0000","last_undegraded":"2023-10-06T01:41:13.348956+0000","last_fullsized":"2023-10-06T01:41:13.348956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:21:30.451862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.355700+0000","last_change":"2023-10-06T01:41:13.350352+0000","last_active":"2023-10-06T01:41:13.355700+0000","last_peered":"2023-10-06T01:41:13.355700+0000","last_clean":"2023-10-06T01:41:13.355700+0000","last_became_active":"2023-10-06T01:41:13.350133+0000","last_became_peered":"2023-10-06T01:41:13.350133+0000","last_unstale":"2023-10-06T01:41:13.355700+0000","last_undegraded":"2023-10-06T01:41:13.355700+0000","last_fullsized":"2023-10-06T01:41:13.355700+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:05:37.961571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354695+0000","last_change":"2023-10-06T01:41:13.353241+0000","last_active":"2023-10-06T01:41:13.354695+0000","last_peered":"2023-10-06T01:41:13.354695+0000","last_clean":"2023-10-06T01:41:13.354695+0000","last_became_active":"2023-10-06T01:41:13.352824+0000","last_became_peered":"2023-10-06T01:41:13.352824+0000","last_unstale":"2023-10-06T01:41:13.354695+0000","last_undegraded":"2023-10-06T01:41:13.354695+0000","last_fullsized":"2023-10-06T01:41:13.354695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:52:48.286318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"19'1","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.348819+0000","last_change":"2023-10-06T01:41:12.339675+0000","last_active":"2023-10-06T01:41:13.348819+0000","last_peered":"2023-10-06T01:41:13.348819+0000","last_clean":"2023-10-06T01:41:13.348819+0000","last_became_active":"2023-10-06T01:41:12.339258+0000","last_became_peered":"2023-10-06T01:41:12.339258+0000","last_unstale":"2023-10-06T01:41:13.348819+0000","last_undegraded":"2023-10-06T01:41:13.348819+0000","last_fullsized":"2023-10-06T01:41:13.348819+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:49:54.658088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.357395+0000","last_change":"2023-10-06T01:41:13.352444+0000","last_active":"2023-10-06T01:41:13.357395+0000","last_peered":"2023-10-06T01:41:13.357395+0000","last_clean":"2023-10-06T01:41:13.357395+0000","last_became_active":"2023-10-06T01:41:13.352134+0000","last_became_peered":"2023-10-06T01:41:13.352134+0000","last_unstale":"2023-10-06T01:41:13.357395+0000","last_undegraded":"2023-10-06T01:41:13.357395+0000","last_fullsized":"2023-10-06T01:41:13.357395+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:11:26.794025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.336930+0000","last_change":"2023-10-06T01:41:12.347654+0000","last_active":"2023-10-06T01:41:13.336930+0000","last_peered":"2023-10-06T01:41:13.336930+0000","last_clean":"2023-10-06T01:41:13.336930+0000","last_became_active":"2023-10-06T01:41:12.347386+0000","last_became_peered":"2023-10-06T01:41:12.347386+0000","last_unstale":"2023-10-06T01:41:13.336930+0000","last_undegraded":"2023-10-06T01:41:13.336930+0000","last_fullsized":"2023-10-06T01:41:13.336930+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:07:30.529679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356475+0000","last_change":"2023-10-06T01:41:13.352350+0000","last_active":"2023-10-06T01:41:13.356475+0000","last_peered":"2023-10-06T01:41:13.356475+0000","last_clean":"2023-10-06T01:41:13.356475+0000","last_became_active":"2023-10-06T01:41:13.352091+0000","last_became_peered":"2023-10-06T01:41:13.352091+0000","last_unstale":"2023-10-06T01:41:13.356475+0000","last_undegraded":"2023-10-06T01:41:13.356475+0000","last_fullsized":"2023-10-06T01:41:13.356475+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:31:20.840764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.334517+0000","last_change":"2023-10-06T01:41:12.343086+0000","last_active":"2023-10-06T01:41:13.334517+0000","last_peered":"2023-10-06T01:41:13.334517+0000","last_clean":"2023-10-06T01:41:13.334517+0000","last_became_active":"2023-10-06T01:41:12.342595+0000","last_became_peered":"2023-10-06T01:41:12.342595+0000","last_unstale":"2023-10-06T01:41:13.334517+0000","last_undegraded":"2023-10-06T01:41:13.334517+0000","last_fullsized":"2023-10-06T01:41:13.334517+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:15:40.841990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351933+0000","last_change":"2023-10-06T01:41:13.347027+0000","last_active":"2023-10-06T01:41:13.351933+0000","last_peered":"2023-10-06T01:41:13.351933+0000","last_clean":"2023-10-06T01:41:13.351933+0000","last_became_active":"2023-10-06T01:41:13.346469+0000","last_became_peered":"2023-10-06T01:41:13.346469+0000","last_unstale":"2023-10-06T01:41:13.351933+0000","last_undegraded":"2023-10-06T01:41:13.351933+0000","last_fullsized":"2023-10-06T01:41:13.351933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:37:14.679660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.329704+0000","last_change":"2023-10-06T01:41:12.341922+0000","last_active":"2023-10-06T01:41:13.329704+0000","last_peered":"2023-10-06T01:41:13.329704+0000","last_clean":"2023-10-06T01:41:13.329704+0000","last_became_active":"2023-10-06T01:41:12.341543+0000","last_became_peered":"2023-10-06T01:41:12.341543+0000","last_unstale":"2023-10-06T01:41:13.329704+0000","last_undegraded":"2023-10-06T01:41:13.329704+0000","last_fullsized":"2023-10-06T01:41:13.329704+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:13:14.004577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352198+0000","last_change":"2023-10-06T01:41:13.345274+0000","last_active":"2023-10-06T01:41:13.352198+0000","last_peered":"2023-10-06T01:41:13.352198+0000","last_clean":"2023-10-06T01:41:13.352198+0000","last_became_active":"2023-10-06T01:41:13.344992+0000","last_became_peered":"2023-10-06T01:41:13.344992+0000","last_unstale":"2023-10-06T01:41:13.352198+0000","last_undegraded":"2023-10-06T01:41:13.352198+0000","last_fullsized":"2023-10-06T01:41:13.352198+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:39:03.091509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.339176+0000","last_change":"2023-10-06T01:41:12.343531+0000","last_active":"2023-10-06T01:41:13.339176+0000","last_peered":"2023-10-06T01:41:13.339176+0000","last_clean":"2023-10-06T01:41:13.339176+0000","last_became_active":"2023-10-06T01:41:12.343256+0000","last_became_peered":"2023-10-06T01:41:12.343256+0000","last_unstale":"2023-10-06T01:41:13.339176+0000","last_undegraded":"2023-10-06T01:41:13.339176+0000","last_fullsized":"2023-10-06T01:41:13.339176+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:16:14.826869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.358071+0000","last_change":"2023-10-06T01:41:13.357361+0000","last_active":"2023-10-06T01:41:13.358071+0000","last_peered":"2023-10-06T01:41:13.358071+0000","last_clean":"2023-10-06T01:41:13.358071+0000","last_became_active":"2023-10-06T01:41:13.356980+0000","last_became_peered":"2023-10-06T01:41:13.356980+0000","last_unstale":"2023-10-06T01:41:13.358071+0000","last_undegraded":"2023-10-06T01:41:13.358071+0000","last_fullsized":"2023-10-06T01:41:13.358071+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:42:51.581939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"19'3","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.355632+0000","last_change":"2023-10-06T01:41:12.350010+0000","last_active":"2023-10-06T01:41:13.355632+0000","last_peered":"2023-10-06T01:41:13.355632+0000","last_clean":"2023-10-06T01:41:13.355632+0000","last_became_active":"2023-10-06T01:41:12.349758+0000","last_became_peered":"2023-10-06T01:41:12.349758+0000","last_unstale":"2023-10-06T01:41:13.355632+0000","last_undegraded":"2023-10-06T01:41:13.355632+0000","last_fullsized":"2023-10-06T01:41:13.355632+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:12:51.734395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352993+0000","last_change":"2023-10-06T01:41:13.348953+0000","last_active":"2023-10-06T01:41:13.352993+0000","last_peered":"2023-10-06T01:41:13.352993+0000","last_clean":"2023-10-06T01:41:13.352993+0000","last_became_active":"2023-10-06T01:41:13.348683+0000","last_became_peered":"2023-10-06T01:41:13.348683+0000","last_unstale":"2023-10-06T01:41:13.352993+0000","last_undegraded":"2023-10-06T01:41:13.352993+0000","last_fullsized":"2023-10-06T01:41:13.352993+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:42:46.214805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"19'6","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.357074+0000","last_change":"2023-10-06T01:41:12.340922+0000","last_active":"2023-10-06T01:41:13.357074+0000","last_peered":"2023-10-06T01:41:13.357074+0000","last_clean":"2023-10-06T01:41:13.357074+0000","last_became_active":"2023-10-06T01:41:12.340598+0000","last_became_peered":"2023-10-06T01:41:12.340598+0000","last_unstale":"2023-10-06T01:41:13.357074+0000","last_undegraded":"2023-10-06T01:41:13.357074+0000","last_fullsized":"2023-10-06T01:41:13.357074+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:06:19.029137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351653+0000","last_change":"2023-10-06T01:41:13.343918+0000","last_active":"2023-10-06T01:41:13.351653+0000","last_peered":"2023-10-06T01:41:13.351653+0000","last_clean":"2023-10-06T01:41:13.351653+0000","last_became_active":"2023-10-06T01:41:13.343621+0000","last_became_peered":"2023-10-06T01:41:13.343621+0000","last_unstale":"2023-10-06T01:41:13.351653+0000","last_undegraded":"2023-10-06T01:41:13.351653+0000","last_fullsized":"2023-10-06T01:41:13.351653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:42:44.799267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.350439+0000","last_change":"2023-10-06T01:41:12.343541+0000","last_active":"2023-10-06T01:41:13.350439+0000","last_peered":"2023-10-06T01:41:13.350439+0000","last_clean":"2023-10-06T01:41:13.350439+0000","last_became_active":"2023-10-06T01:41:12.342645+0000","last_became_peered":"2023-10-06T01:41:12.342645+0000","last_unstale":"2023-10-06T01:41:13.350439+0000","last_undegraded":"2023-10-06T01:41:13.350439+0000","last_fullsized":"2023-10-06T01:41:13.350439+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:16:14.826869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.357594+0000","last_change":"2023-10-06T01:41:13.352574+0000","last_active":"2023-10-06T01:41:13.357594+0000","last_peered":"2023-10-06T01:41:13.357594+0000","last_clean":"2023-10-06T01:41:13.357594+0000","last_became_active":"2023-10-06T01:41:13.352300+0000","last_became_peered":"2023-10-06T01:41:13.352300+0000","last_unstale":"2023-10-06T01:41:13.357594+0000","last_undegraded":"2023-10-06T01:41:13.357594+0000","last_fullsized":"2023-10-06T01:41:13.357594+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:27:54.074728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.329224+0000","last_change":"2023-10-06T01:41:12.344679+0000","last_active":"2023-10-06T01:41:13.329224+0000","last_peered":"2023-10-06T01:41:13.329224+0000","last_clean":"2023-10-06T01:41:13.329224+0000","last_became_active":"2023-10-06T01:41:12.344430+0000","last_became_peered":"2023-10-06T01:41:12.344430+0000","last_unstale":"2023-10-06T01:41:13.329224+0000","last_undegraded":"2023-10-06T01:41:13.329224+0000","last_fullsized":"2023-10-06T01:41:13.329224+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:27:01.582307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.360786+0000","last_change":"2023-10-06T01:41:13.357148+0000","last_active":"2023-10-06T01:41:13.360786+0000","last_peered":"2023-10-06T01:41:13.360786+0000","last_clean":"2023-10-06T01:41:13.360786+0000","last_became_active":"2023-10-06T01:41:13.356843+0000","last_became_peered":"2023-10-06T01:41:13.356843+0000","last_unstale":"2023-10-06T01:41:13.360786+0000","last_undegraded":"2023-10-06T01:41:13.360786+0000","last_fullsized":"2023-10-06T01:41:13.360786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T01:51:57.229301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.334167+0000","last_change":"2023-10-06T01:41:12.344068+0000","last_active":"2023-10-06T01:41:13.334167+0000","last_peered":"2023-10-06T01:41:13.334167+0000","last_clean":"2023-10-06T01:41:13.334167+0000","last_became_active":"2023-10-06T01:41:12.342562+0000","last_became_peered":"2023-10-06T01:41:12.342562+0000","last_unstale":"2023-10-06T01:41:13.334167+0000","last_undegraded":"2023-10-06T01:41:13.334167+0000","last_fullsized":"2023-10-06T01:41:13.334167+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:37:13.672716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.346441+0000","last_change":"2023-10-06T01:41:13.343508+0000","last_active":"2023-10-06T01:41:13.346441+0000","last_peered":"2023-10-06T01:41:13.346441+0000","last_clean":"2023-10-06T01:41:13.346441+0000","last_became_active":"2023-10-06T01:41:13.343249+0000","last_became_peered":"2023-10-06T01:41:13.343249+0000","last_unstale":"2023-10-06T01:41:13.346441+0000","last_undegraded":"2023-10-06T01:41:13.346441+0000","last_fullsized":"2023-10-06T01:41:13.346441+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:03:37.385158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"19'8","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.347028+0000","last_change":"2023-10-06T01:41:12.342949+0000","last_active":"2023-10-06T01:41:13.347028+0000","last_peered":"2023-10-06T01:41:13.347028+0000","last_clean":"2023-10-06T01:41:13.347028+0000","last_became_active":"2023-10-06T01:41:12.342687+0000","last_became_peered":"2023-10-06T01:41:12.342687+0000","last_unstale":"2023-10-06T01:41:13.347028+0000","last_undegraded":"2023-10-06T01:41:13.347028+0000","last_fullsized":"2023-10-06T01:41:13.347028+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:37:52.143727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.357895+0000","last_change":"2023-10-06T01:41:13.354861+0000","last_active":"2023-10-06T01:41:13.357895+0000","last_peered":"2023-10-06T01:41:13.357895+0000","last_clean":"2023-10-06T01:41:13.357895+0000","last_became_active":"2023-10-06T01:41:13.354190+0000","last_became_peered":"2023-10-06T01:41:13.354190+0000","last_unstale":"2023-10-06T01:41:13.357895+0000","last_undegraded":"2023-10-06T01:41:13.357895+0000","last_fullsized":"2023-10-06T01:41:13.357895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:45:07.571069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"19'1","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.349986+0000","last_change":"2023-10-06T01:41:12.344048+0000","last_active":"2023-10-06T01:41:13.349986+0000","last_peered":"2023-10-06T01:41:13.349986+0000","last_clean":"2023-10-06T01:41:13.349986+0000","last_became_active":"2023-10-06T01:41:12.343532+0000","last_became_peered":"2023-10-06T01:41:12.343532+0000","last_unstale":"2023-10-06T01:41:13.349986+0000","last_undegraded":"2023-10-06T01:41:13.349986+0000","last_fullsized":"2023-10-06T01:41:13.349986+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:50:44.552768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354457+0000","last_change":"2023-10-06T01:41:13.351276+0000","last_active":"2023-10-06T01:41:13.354457+0000","last_peered":"2023-10-06T01:41:13.354457+0000","last_clean":"2023-10-06T01:41:13.354457+0000","last_became_active":"2023-10-06T01:41:13.351005+0000","last_became_peered":"2023-10-06T01:41:13.351005+0000","last_unstale":"2023-10-06T01:41:13.354457+0000","last_undegraded":"2023-10-06T01:41:13.354457+0000","last_fullsized":"2023-10-06T01:41:13.354457+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:42:51.581939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356755+0000","last_change":"2023-10-06T01:41:12.344159+0000","last_active":"2023-10-06T01:41:13.356755+0000","last_peered":"2023-10-06T01:41:13.356755+0000","last_clean":"2023-10-06T01:41:13.356755+0000","last_became_active":"2023-10-06T01:41:12.343593+0000","last_became_peered":"2023-10-06T01:41:12.343593+0000","last_unstale":"2023-10-06T01:41:13.356755+0000","last_undegraded":"2023-10-06T01:41:13.356755+0000","last_fullsized":"2023-10-06T01:41:13.356755+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:27:01.582307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354635+0000","last_change":"2023-10-06T01:41:13.349120+0000","last_active":"2023-10-06T01:41:13.354635+0000","last_peered":"2023-10-06T01:41:13.354635+0000","last_clean":"2023-10-06T01:41:13.354635+0000","last_became_active":"2023-10-06T01:41:13.348850+0000","last_became_peered":"2023-10-06T01:41:13.348850+0000","last_unstale":"2023-10-06T01:41:13.354635+0000","last_undegraded":"2023-10-06T01:41:13.354635+0000","last_fullsized":"2023-10-06T01:41:13.354635+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:27:32.273170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"19'1","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.358476+0000","last_change":"2023-10-06T01:41:12.344880+0000","last_active":"2023-10-06T01:41:13.358476+0000","last_peered":"2023-10-06T01:41:13.358476+0000","last_clean":"2023-10-06T01:41:13.358476+0000","last_became_active":"2023-10-06T01:41:12.344676+0000","last_became_peered":"2023-10-06T01:41:12.344676+0000","last_unstale":"2023-10-06T01:41:13.358476+0000","last_undegraded":"2023-10-06T01:41:13.358476+0000","last_fullsized":"2023-10-06T01:41:13.358476+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:21:09.833061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.353951+0000","last_change":"2023-10-06T01:41:13.348117+0000","last_active":"2023-10-06T01:41:13.353951+0000","last_peered":"2023-10-06T01:41:13.353951+0000","last_clean":"2023-10-06T01:41:13.353951+0000","last_became_active":"2023-10-06T01:41:13.347872+0000","last_became_peered":"2023-10-06T01:41:13.347872+0000","last_unstale":"2023-10-06T01:41:13.353951+0000","last_undegraded":"2023-10-06T01:41:13.353951+0000","last_fullsized":"2023-10-06T01:41:13.353951+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:21:10.840006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.338677+0000","last_change":"2023-10-06T01:41:12.341362+0000","last_active":"2023-10-06T01:41:13.338677+0000","last_peered":"2023-10-06T01:41:13.338677+0000","last_clean":"2023-10-06T01:41:13.338677+0000","last_became_active":"2023-10-06T01:41:12.340855+0000","last_became_peered":"2023-10-06T01:41:12.340855+0000","last_unstale":"2023-10-06T01:41:13.338677+0000","last_undegraded":"2023-10-06T01:41:13.338677+0000","last_fullsized":"2023-10-06T01:41:13.338677+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:15:40.841990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.349189+0000","last_change":"2023-10-06T01:41:13.344966+0000","last_active":"2023-10-06T01:41:13.349189+0000","last_peered":"2023-10-06T01:41:13.349189+0000","last_clean":"2023-10-06T01:41:13.349189+0000","last_became_active":"2023-10-06T01:41:13.344700+0000","last_became_peered":"2023-10-06T01:41:13.344700+0000","last_unstale":"2023-10-06T01:41:13.349189+0000","last_undegraded":"2023-10-06T01:41:13.349189+0000","last_fullsized":"2023-10-06T01:41:13.349189+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:42:23.199959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.346337+0000","last_change":"2023-10-06T01:41:12.343169+0000","last_active":"2023-10-06T01:41:13.346337+0000","last_peered":"2023-10-06T01:41:13.346337+0000","last_clean":"2023-10-06T01:41:13.346337+0000","last_became_active":"2023-10-06T01:41:12.342474+0000","last_became_peered":"2023-10-06T01:41:12.342474+0000","last_unstale":"2023-10-06T01:41:13.346337+0000","last_undegraded":"2023-10-06T01:41:13.346337+0000","last_fullsized":"2023-10-06T01:41:13.346337+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:27:01.582307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351608+0000","last_change":"2023-10-06T01:41:13.343671+0000","last_active":"2023-10-06T01:41:13.351608+0000","last_peered":"2023-10-06T01:41:13.351608+0000","last_clean":"2023-10-06T01:41:13.351608+0000","last_became_active":"2023-10-06T01:41:13.341868+0000","last_became_peered":"2023-10-06T01:41:13.341868+0000","last_unstale":"2023-10-06T01:41:13.351608+0000","last_undegraded":"2023-10-06T01:41:13.351608+0000","last_fullsized":"2023-10-06T01:41:13.351608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T01:51:57.229301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.329621+0000","last_change":"2023-10-06T01:41:12.339878+0000","last_active":"2023-10-06T01:41:13.329621+0000","last_peered":"2023-10-06T01:41:13.329621+0000","last_clean":"2023-10-06T01:41:13.329621+0000","last_became_active":"2023-10-06T01:41:12.338935+0000","last_became_peered":"2023-10-06T01:41:12.338935+0000","last_unstale":"2023-10-06T01:41:13.329621+0000","last_undegraded":"2023-10-06T01:41:13.329621+0000","last_fullsized":"2023-10-06T01:41:13.329621+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:44:53.014481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351640+0000","last_change":"2023-10-06T01:41:13.344292+0000","last_active":"2023-10-06T01:41:13.351640+0000","last_peered":"2023-10-06T01:41:13.351640+0000","last_clean":"2023-10-06T01:41:13.351640+0000","last_became_active":"2023-10-06T01:41:13.343947+0000","last_became_peered":"2023-10-06T01:41:13.343947+0000","last_unstale":"2023-10-06T01:41:13.351640+0000","last_undegraded":"2023-10-06T01:41:13.351640+0000","last_fullsized":"2023-10-06T01:41:13.351640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:45:07.571069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.331713+0000","last_change":"2023-10-06T01:41:12.336355+0000","last_active":"2023-10-06T01:41:13.331713+0000","last_peered":"2023-10-06T01:41:13.331713+0000","last_clean":"2023-10-06T01:41:13.331713+0000","last_became_active":"2023-10-06T01:41:12.336064+0000","last_became_peered":"2023-10-06T01:41:12.336064+0000","last_unstale":"2023-10-06T01:41:13.331713+0000","last_undegraded":"2023-10-06T01:41:13.331713+0000","last_fullsized":"2023-10-06T01:41:13.331713+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:16:14.826869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354641+0000","last_change":"2023-10-06T01:41:13.346404+0000","last_active":"2023-10-06T01:41:13.354641+0000","last_peered":"2023-10-06T01:41:13.354641+0000","last_clean":"2023-10-06T01:41:13.354641+0000","last_became_active":"2023-10-06T01:41:13.343805+0000","last_became_peered":"2023-10-06T01:41:13.343805+0000","last_unstale":"2023-10-06T01:41:13.354641+0000","last_undegraded":"2023-10-06T01:41:13.354641+0000","last_fullsized":"2023-10-06T01:41:13.354641+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:11:26.794025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.334388+0000","last_change":"2023-10-06T01:41:12.345661+0000","last_active":"2023-10-06T01:41:13.334388+0000","last_peered":"2023-10-06T01:41:13.334388+0000","last_clean":"2023-10-06T01:41:13.334388+0000","last_became_active":"2023-10-06T01:41:12.345363+0000","last_became_peered":"2023-10-06T01:41:12.345363+0000","last_unstale":"2023-10-06T01:41:13.334388+0000","last_undegraded":"2023-10-06T01:41:13.334388+0000","last_fullsized":"2023-10-06T01:41:13.334388+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:50:44.552768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356650+0000","last_change":"2023-10-06T01:41:13.353633+0000","last_active":"2023-10-06T01:41:13.356650+0000","last_peered":"2023-10-06T01:41:13.356650+0000","last_clean":"2023-10-06T01:41:13.356650+0000","last_became_active":"2023-10-06T01:41:13.353318+0000","last_became_peered":"2023-10-06T01:41:13.353318+0000","last_unstale":"2023-10-06T01:41:13.356650+0000","last_undegraded":"2023-10-06T01:41:13.356650+0000","last_fullsized":"2023-10-06T01:41:13.356650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:42:23.199959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356615+0000","last_change":"2023-10-06T01:41:12.340463+0000","last_active":"2023-10-06T01:41:13.356615+0000","last_peered":"2023-10-06T01:41:13.356615+0000","last_clean":"2023-10-06T01:41:13.356615+0000","last_became_active":"2023-10-06T01:41:12.340186+0000","last_became_peered":"2023-10-06T01:41:12.340186+0000","last_unstale":"2023-10-06T01:41:13.356615+0000","last_undegraded":"2023-10-06T01:41:13.356615+0000","last_fullsized":"2023-10-06T01:41:13.356615+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:16:14.826869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.330869+0000","last_change":"2023-10-06T01:41:07.454295+0000","last_active":"2023-10-06T01:41:13.330869+0000","last_peered":"2023-10-06T01:41:13.330869+0000","last_clean":"2023-10-06T01:41:13.330869+0000","last_became_active":"2023-10-06T01:41:07.454034+0000","last_became_peered":"2023-10-06T01:41:07.454034+0000","last_unstale":"2023-10-06T01:41:13.330869+0000","last_undegraded":"2023-10-06T01:41:13.330869+0000","last_fullsized":"2023-10-06T01:41:13.330869+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:32:58.914183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354272+0000","last_change":"2023-10-06T01:41:13.351289+0000","last_active":"2023-10-06T01:41:13.354272+0000","last_peered":"2023-10-06T01:41:13.354272+0000","last_clean":"2023-10-06T01:41:13.354272+0000","last_became_active":"2023-10-06T01:41:13.351022+0000","last_became_peered":"2023-10-06T01:41:13.351022+0000","last_unstale":"2023-10-06T01:41:13.354272+0000","last_undegraded":"2023-10-06T01:41:13.354272+0000","last_fullsized":"2023-10-06T01:41:13.354272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T01:51:57.229301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.334655+0000","last_change":"2023-10-06T01:41:07.261955+0000","last_active":"2023-10-06T01:41:13.334655+0000","last_peered":"2023-10-06T01:41:13.334655+0000","last_clean":"2023-10-06T01:41:13.334655+0000","last_became_active":"2023-10-06T01:41:07.261692+0000","last_became_peered":"2023-10-06T01:41:07.261692+0000","last_unstale":"2023-10-06T01:41:13.334655+0000","last_undegraded":"2023-10-06T01:41:13.334655+0000","last_fullsized":"2023-10-06T01:41:13.334655+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"20'1","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-06T01:41:15.784088+0000","last_change":"2023-10-06T01:41:12.341312+0000","last_active":"2023-10-06T01:41:15.784088+0000","last_peered":"2023-10-06T01:41:15.784088+0000","last_clean":"2023-10-06T01:41:15.784088+0000","last_became_active":"2023-10-06T01:41:12.340997+0000","last_became_peered":"2023-10-06T01:41:12.340997+0000","last_unstale":"2023-10-06T01:41:15.784088+0000","last_undegraded":"2023-10-06T01:41:15.784088+0000","last_fullsized":"2023-10-06T01:41:15.784088+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:06:19.029137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351571+0000","last_change":"2023-10-06T01:41:13.344156+0000","last_active":"2023-10-06T01:41:13.351571+0000","last_peered":"2023-10-06T01:41:13.351571+0000","last_clean":"2023-10-06T01:41:13.351571+0000","last_became_active":"2023-10-06T01:41:13.343601+0000","last_became_peered":"2023-10-06T01:41:13.343601+0000","last_unstale":"2023-10-06T01:41:13.351571+0000","last_undegraded":"2023-10-06T01:41:13.351571+0000","last_fullsized":"2023-10-06T01:41:13.351571+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:21:42.851681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.336375+0000","last_change":"2023-10-06T01:41:07.258362+0000","last_active":"2023-10-06T01:41:13.336375+0000","last_peered":"2023-10-06T01:41:13.336375+0000","last_clean":"2023-10-06T01:41:13.336375+0000","last_became_active":"2023-10-06T01:41:07.258055+0000","last_became_peered":"2023-10-06T01:41:07.258055+0000","last_unstale":"2023-10-06T01:41:13.336375+0000","last_undegraded":"2023-10-06T01:41:13.336375+0000","last_fullsized":"2023-10-06T01:41:13.336375+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"16'117","reported_seq":163,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.334859+0000","last_change":"2023-10-06T01:41:07.261164+0000","last_active":"2023-10-06T01:41:13.334859+0000","last_peered":"2023-10-06T01:41:13.334859+0000","last_clean":"2023-10-06T01:41:13.334859+0000","last_became_active":"2023-10-06T01:41:07.260896+0000","last_became_peered":"2023-10-06T01:41:07.260896+0000","last_unstale":"2023-10-06T01:41:13.334859+0000","last_undegraded":"2023-10-06T01:41:13.334859+0000","last_fullsized":"2023-10-06T01:41:13.334859+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:05:22.934401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.328964+0000","last_change":"2023-10-06T01:41:12.343057+0000","last_active":"2023-10-06T01:41:13.328964+0000","last_peered":"2023-10-06T01:41:13.328964+0000","last_clean":"2023-10-06T01:41:13.328964+0000","last_became_active":"2023-10-06T01:41:12.342793+0000","last_became_peered":"2023-10-06T01:41:12.342793+0000","last_unstale":"2023-10-06T01:41:13.328964+0000","last_undegraded":"2023-10-06T01:41:13.328964+0000","last_fullsized":"2023-10-06T01:41:13.328964+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:15:40.841990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351969+0000","last_change":"2023-10-06T01:41:13.346571+0000","last_active":"2023-10-06T01:41:13.351969+0000","last_peered":"2023-10-06T01:41:13.351969+0000","last_clean":"2023-10-06T01:41:13.351969+0000","last_became_active":"2023-10-06T01:41:13.346077+0000","last_became_peered":"2023-10-06T01:41:13.346077+0000","last_unstale":"2023-10-06T01:41:13.351969+0000","last_undegraded":"2023-10-06T01:41:13.351969+0000","last_fullsized":"2023-10-06T01:41:13.351969+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:42:51.581939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.333951+0000","last_change":"2023-10-06T01:41:07.456511+0000","last_active":"2023-10-06T01:41:13.333951+0000","last_peered":"2023-10-06T01:41:13.333951+0000","last_clean":"2023-10-06T01:41:13.333951+0000","last_became_active":"2023-10-06T01:41:07.456024+0000","last_became_peered":"2023-10-06T01:41:07.456024+0000","last_unstale":"2023-10-06T01:41:13.333951+0000","last_undegraded":"2023-10-06T01:41:13.333951+0000","last_fullsized":"2023-10-06T01:41:13.333951+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:51:02.949231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.336635+0000","last_change":"2023-10-06T01:41:12.343382+0000","last_active":"2023-10-06T01:41:13.336635+0000","last_peered":"2023-10-06T01:41:13.336635+0000","last_clean":"2023-10-06T01:41:13.336635+0000","last_became_active":"2023-10-06T01:41:12.343088+0000","last_became_peered":"2023-10-06T01:41:12.343088+0000","last_unstale":"2023-10-06T01:41:13.336635+0000","last_undegraded":"2023-10-06T01:41:13.336635+0000","last_fullsized":"2023-10-06T01:41:13.336635+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:44:53.014481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.350157+0000","last_change":"2023-10-06T01:41:13.346852+0000","last_active":"2023-10-06T01:41:13.350157+0000","last_peered":"2023-10-06T01:41:13.350157+0000","last_clean":"2023-10-06T01:41:13.350157+0000","last_became_active":"2023-10-06T01:41:13.346571+0000","last_became_peered":"2023-10-06T01:41:13.346571+0000","last_unstale":"2023-10-06T01:41:13.350157+0000","last_undegraded":"2023-10-06T01:41:13.350157+0000","last_fullsized":"2023-10-06T01:41:13.350157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:25:04.563235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.334453+0000","last_change":"2023-10-06T01:41:07.456273+0000","last_active":"2023-10-06T01:41:13.334453+0000","last_peered":"2023-10-06T01:41:13.334453+0000","last_clean":"2023-10-06T01:41:13.334453+0000","last_became_active":"2023-10-06T01:41:07.456000+0000","last_became_peered":"2023-10-06T01:41:07.456000+0000","last_unstale":"2023-10-06T01:41:13.334453+0000","last_undegraded":"2023-10-06T01:41:13.334453+0000","last_fullsized":"2023-10-06T01:41:13.334453+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:01:42.633388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.334637+0000","last_change":"2023-10-06T01:41:12.342215+0000","last_active":"2023-10-06T01:41:13.334637+0000","last_peered":"2023-10-06T01:41:13.334637+0000","last_clean":"2023-10-06T01:41:13.334637+0000","last_became_active":"2023-10-06T01:41:12.341493+0000","last_became_peered":"2023-10-06T01:41:12.341493+0000","last_unstale":"2023-10-06T01:41:13.334637+0000","last_undegraded":"2023-10-06T01:41:13.334637+0000","last_fullsized":"2023-10-06T01:41:13.334637+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:49:54.658088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352134+0000","last_change":"2023-10-06T01:41:13.346680+0000","last_active":"2023-10-06T01:41:13.352134+0000","last_peered":"2023-10-06T01:41:13.352134+0000","last_clean":"2023-10-06T01:41:13.352134+0000","last_became_active":"2023-10-06T01:41:13.346326+0000","last_became_peered":"2023-10-06T01:41:13.346326+0000","last_unstale":"2023-10-06T01:41:13.352134+0000","last_undegraded":"2023-10-06T01:41:13.352134+0000","last_fullsized":"2023-10-06T01:41:13.352134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:12:52.741339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.338955+0000","last_change":"2023-10-06T01:41:07.456688+0000","last_active":"2023-10-06T01:41:13.338955+0000","last_peered":"2023-10-06T01:41:13.338955+0000","last_clean":"2023-10-06T01:41:13.338955+0000","last_became_active":"2023-10-06T01:41:07.456304+0000","last_became_peered":"2023-10-06T01:41:07.456304+0000","last_unstale":"2023-10-06T01:41:13.338955+0000","last_undegraded":"2023-10-06T01:41:13.338955+0000","last_fullsized":"2023-10-06T01:41:13.338955+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.332203+0000","last_change":"2023-10-06T01:41:12.342354+0000","last_active":"2023-10-06T01:41:13.332203+0000","last_peered":"2023-10-06T01:41:13.332203+0000","last_clean":"2023-10-06T01:41:13.332203+0000","last_became_active":"2023-10-06T01:41:12.342066+0000","last_became_peered":"2023-10-06T01:41:12.342066+0000","last_unstale":"2023-10-06T01:41:13.332203+0000","last_undegraded":"2023-10-06T01:41:13.332203+0000","last_fullsized":"2023-10-06T01:41:13.332203+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:15:40.841990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352905+0000","last_change":"2023-10-06T01:41:13.348241+0000","last_active":"2023-10-06T01:41:13.352905+0000","last_peered":"2023-10-06T01:41:13.352905+0000","last_clean":"2023-10-06T01:41:13.352905+0000","last_became_active":"2023-10-06T01:41:13.347979+0000","last_became_peered":"2023-10-06T01:41:13.347979+0000","last_unstale":"2023-10-06T01:41:13.352905+0000","last_undegraded":"2023-10-06T01:41:13.352905+0000","last_fullsized":"2023-10-06T01:41:13.352905+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:41:59.186469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.329416+0000","last_change":"2023-10-06T01:41:07.456388+0000","last_active":"2023-10-06T01:41:13.329416+0000","last_peered":"2023-10-06T01:41:13.329416+0000","last_clean":"2023-10-06T01:41:13.329416+0000","last_became_active":"2023-10-06T01:41:07.456045+0000","last_became_peered":"2023-10-06T01:41:07.456045+0000","last_unstale":"2023-10-06T01:41:13.329416+0000","last_undegraded":"2023-10-06T01:41:13.329416+0000","last_fullsized":"2023-10-06T01:41:13.329416+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.338387+0000","last_change":"2023-10-06T01:41:12.344268+0000","last_active":"2023-10-06T01:41:13.338387+0000","last_peered":"2023-10-06T01:41:13.338387+0000","last_clean":"2023-10-06T01:41:13.338387+0000","last_became_active":"2023-10-06T01:41:12.343996+0000","last_became_peered":"2023-10-06T01:41:12.343996+0000","last_unstale":"2023-10-06T01:41:13.338387+0000","last_undegraded":"2023-10-06T01:41:13.338387+0000","last_fullsized":"2023-10-06T01:41:13.338387+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:37:52.143727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351761+0000","last_change":"2023-10-06T01:41:13.346184+0000","last_active":"2023-10-06T01:41:13.351761+0000","last_peered":"2023-10-06T01:41:13.351761+0000","last_clean":"2023-10-06T01:41:13.351761+0000","last_became_active":"2023-10-06T01:41:13.345570+0000","last_became_peered":"2023-10-06T01:41:13.345570+0000","last_unstale":"2023-10-06T01:41:13.351761+0000","last_undegraded":"2023-10-06T01:41:13.351761+0000","last_fullsized":"2023-10-06T01:41:13.351761+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:42:46.214805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.330470+0000","last_change":"2023-10-06T01:41:07.258162+0000","last_active":"2023-10-06T01:41:13.330470+0000","last_peered":"2023-10-06T01:41:13.330470+0000","last_clean":"2023-10-06T01:41:13.330470+0000","last_became_active":"2023-10-06T01:41:07.257795+0000","last_became_peered":"2023-10-06T01:41:07.257795+0000","last_unstale":"2023-10-06T01:41:13.330470+0000","last_undegraded":"2023-10-06T01:41:13.330470+0000","last_fullsized":"2023-10-06T01:41:13.330470+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:05:22.934401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.337002+0000","last_change":"2023-10-06T01:41:12.341990+0000","last_active":"2023-10-06T01:41:13.337002+0000","last_peered":"2023-10-06T01:41:13.337002+0000","last_clean":"2023-10-06T01:41:13.337002+0000","last_became_active":"2023-10-06T01:41:12.341680+0000","last_became_peered":"2023-10-06T01:41:12.341680+0000","last_unstale":"2023-10-06T01:41:13.337002+0000","last_undegraded":"2023-10-06T01:41:13.337002+0000","last_fullsized":"2023-10-06T01:41:13.337002+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:50:44.552768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.349056+0000","last_change":"2023-10-06T01:41:13.344589+0000","last_active":"2023-10-06T01:41:13.349056+0000","last_peered":"2023-10-06T01:41:13.349056+0000","last_clean":"2023-10-06T01:41:13.349056+0000","last_became_active":"2023-10-06T01:41:13.344281+0000","last_became_peered":"2023-10-06T01:41:13.344281+0000","last_unstale":"2023-10-06T01:41:13.349056+0000","last_undegraded":"2023-10-06T01:41:13.349056+0000","last_fullsized":"2023-10-06T01:41:13.349056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:32:17.938309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"19'1","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.345810+0000","last_change":"2023-10-06T01:41:12.347655+0000","last_active":"2023-10-06T01:41:13.345810+0000","last_peered":"2023-10-06T01:41:13.345810+0000","last_clean":"2023-10-06T01:41:13.345810+0000","last_became_active":"2023-10-06T01:41:12.347353+0000","last_became_peered":"2023-10-06T01:41:12.347353+0000","last_unstale":"2023-10-06T01:41:13.345810+0000","last_undegraded":"2023-10-06T01:41:13.345810+0000","last_fullsized":"2023-10-06T01:41:13.345810+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:49:54.658088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352446+0000","last_change":"2023-10-06T01:41:13.349874+0000","last_active":"2023-10-06T01:41:13.352446+0000","last_peered":"2023-10-06T01:41:13.352446+0000","last_clean":"2023-10-06T01:41:13.352446+0000","last_became_active":"2023-10-06T01:41:13.349607+0000","last_became_peered":"2023-10-06T01:41:13.349607+0000","last_unstale":"2023-10-06T01:41:13.352446+0000","last_undegraded":"2023-10-06T01:41:13.352446+0000","last_fullsized":"2023-10-06T01:41:13.352446+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:06:20.036081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352575+0000","last_change":"2023-10-06T01:41:12.341511+0000","last_active":"2023-10-06T01:41:13.352575+0000","last_peered":"2023-10-06T01:41:13.352575+0000","last_clean":"2023-10-06T01:41:13.352575+0000","last_became_active":"2023-10-06T01:41:12.341216+0000","last_became_peered":"2023-10-06T01:41:12.341216+0000","last_unstale":"2023-10-06T01:41:13.352575+0000","last_undegraded":"2023-10-06T01:41:13.352575+0000","last_fullsized":"2023-10-06T01:41:13.352575+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:27:01.582307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351635+0000","last_change":"2023-10-06T01:41:13.343780+0000","last_active":"2023-10-06T01:41:13.351635+0000","last_peered":"2023-10-06T01:41:13.351635+0000","last_clean":"2023-10-06T01:41:13.351635+0000","last_became_active":"2023-10-06T01:41:13.343515+0000","last_became_peered":"2023-10-06T01:41:13.343515+0000","last_unstale":"2023-10-06T01:41:13.351635+0000","last_undegraded":"2023-10-06T01:41:13.351635+0000","last_fullsized":"2023-10-06T01:41:13.351635+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:38:58.407819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.353702+0000","last_change":"2023-10-06T01:41:12.341780+0000","last_active":"2023-10-06T01:41:13.353702+0000","last_peered":"2023-10-06T01:41:13.353702+0000","last_clean":"2023-10-06T01:41:13.353702+0000","last_became_active":"2023-10-06T01:41:12.340585+0000","last_became_peered":"2023-10-06T01:41:12.340585+0000","last_unstale":"2023-10-06T01:41:13.353702+0000","last_undegraded":"2023-10-06T01:41:13.353702+0000","last_fullsized":"2023-10-06T01:41:13.353702+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:49:54.658088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354558+0000","last_change":"2023-10-06T01:41:13.345543+0000","last_active":"2023-10-06T01:41:13.354558+0000","last_peered":"2023-10-06T01:41:13.354558+0000","last_clean":"2023-10-06T01:41:13.354558+0000","last_became_active":"2023-10-06T01:41:13.345286+0000","last_became_peered":"2023-10-06T01:41:13.345286+0000","last_unstale":"2023-10-06T01:41:13.354558+0000","last_undegraded":"2023-10-06T01:41:13.354558+0000","last_fullsized":"2023-10-06T01:41:13.354558+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:52:48.286318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"19'1","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351829+0000","last_change":"2023-10-06T01:41:12.340776+0000","last_active":"2023-10-06T01:41:13.351829+0000","last_peered":"2023-10-06T01:41:13.351829+0000","last_clean":"2023-10-06T01:41:13.351829+0000","last_became_active":"2023-10-06T01:41:12.340294+0000","last_became_peered":"2023-10-06T01:41:12.340294+0000","last_unstale":"2023-10-06T01:41:13.351829+0000","last_undegraded":"2023-10-06T01:41:13.351829+0000","last_fullsized":"2023-10-06T01:41:13.351829+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:37:13.672716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.357969+0000","last_change":"2023-10-06T01:41:13.355273+0000","last_active":"2023-10-06T01:41:13.357969+0000","last_peered":"2023-10-06T01:41:13.357969+0000","last_clean":"2023-10-06T01:41:13.357969+0000","last_became_active":"2023-10-06T01:41:13.354977+0000","last_became_peered":"2023-10-06T01:41:13.354977+0000","last_unstale":"2023-10-06T01:41:13.357969+0000","last_undegraded":"2023-10-06T01:41:13.357969+0000","last_fullsized":"2023-10-06T01:41:13.357969+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:21:42.851681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"19'4","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.345580+0000","last_change":"2023-10-06T01:41:12.338431+0000","last_active":"2023-10-06T01:41:13.345580+0000","last_peered":"2023-10-06T01:41:13.345580+0000","last_clean":"2023-10-06T01:41:13.345580+0000","last_became_active":"2023-10-06T01:41:12.338081+0000","last_became_peered":"2023-10-06T01:41:12.338081+0000","last_unstale":"2023-10-06T01:41:13.345580+0000","last_undegraded":"2023-10-06T01:41:13.345580+0000","last_fullsized":"2023-10-06T01:41:13.345580+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:16:14.826869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351157+0000","last_change":"2023-10-06T01:41:13.342212+0000","last_active":"2023-10-06T01:41:13.351157+0000","last_peered":"2023-10-06T01:41:13.351157+0000","last_clean":"2023-10-06T01:41:13.351157+0000","last_became_active":"2023-10-06T01:41:13.341948+0000","last_became_peered":"2023-10-06T01:41:13.341948+0000","last_unstale":"2023-10-06T01:41:13.351157+0000","last_undegraded":"2023-10-06T01:41:13.351157+0000","last_fullsized":"2023-10-06T01:41:13.351157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:18:23.688664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354188+0000","last_change":"2023-10-06T01:41:13.348482+0000","last_active":"2023-10-06T01:41:13.354188+0000","last_peered":"2023-10-06T01:41:13.354188+0000","last_clean":"2023-10-06T01:41:13.354188+0000","last_became_active":"2023-10-06T01:41:13.348075+0000","last_became_peered":"2023-10-06T01:41:13.348075+0000","last_unstale":"2023-10-06T01:41:13.354188+0000","last_undegraded":"2023-10-06T01:41:13.354188+0000","last_fullsized":"2023-10-06T01:41:13.354188+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:25:04.563235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.357958+0000","last_change":"2023-10-06T01:41:13.355216+0000","last_active":"2023-10-06T01:41:13.357958+0000","last_peered":"2023-10-06T01:41:13.357958+0000","last_clean":"2023-10-06T01:41:13.357958+0000","last_became_active":"2023-10-06T01:41:13.354853+0000","last_became_peered":"2023-10-06T01:41:13.354853+0000","last_unstale":"2023-10-06T01:41:13.357958+0000","last_undegraded":"2023-10-06T01:41:13.357958+0000","last_fullsized":"2023-10-06T01:41:13.357958+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:42:44.799267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351163+0000","last_change":"2023-10-06T01:41:13.342888+0000","last_active":"2023-10-06T01:41:13.351163+0000","last_peered":"2023-10-06T01:41:13.351163+0000","last_clean":"2023-10-06T01:41:13.351163+0000","last_became_active":"2023-10-06T01:41:13.342626+0000","last_became_peered":"2023-10-06T01:41:13.342626+0000","last_unstale":"2023-10-06T01:41:13.351163+0000","last_undegraded":"2023-10-06T01:41:13.351163+0000","last_fullsized":"2023-10-06T01:41:13.351163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:43:29.491858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.357705+0000","last_change":"2023-10-06T01:41:13.353822+0000","last_active":"2023-10-06T01:41:13.357705+0000","last_peered":"2023-10-06T01:41:13.357705+0000","last_clean":"2023-10-06T01:41:13.357705+0000","last_became_active":"2023-10-06T01:41:13.353292+0000","last_became_peered":"2023-10-06T01:41:13.353292+0000","last_unstale":"2023-10-06T01:41:13.357705+0000","last_undegraded":"2023-10-06T01:41:13.357705+0000","last_fullsized":"2023-10-06T01:41:13.357705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:52:48.286318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356636+0000","last_change":"2023-10-06T01:41:13.353662+0000","last_active":"2023-10-06T01:41:13.356636+0000","last_peered":"2023-10-06T01:41:13.356636+0000","last_clean":"2023-10-06T01:41:13.356636+0000","last_became_active":"2023-10-06T01:41:13.353269+0000","last_became_peered":"2023-10-06T01:41:13.353269+0000","last_unstale":"2023-10-06T01:41:13.356636+0000","last_undegraded":"2023-10-06T01:41:13.356636+0000","last_fullsized":"2023-10-06T01:41:13.356636+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:11:44.715827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298096,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073744,"statfs":{"total":96636764160,"available":96331513856,"internally_reserved":0,"allocated":827392,"data_stored":666304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297484,"kb_used_data":196,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074356,"statfs":{"total":96636764160,"available":96332140544,"internally_reserved":0,"allocated":200704,"data_stored":72831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47099999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298068,"kb_used_data":780,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073772,"statfs":{"total":96636764160,"available":96331542528,"internally_reserved":0,"allocated":798720,"data_stored":664552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55700000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297488,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074352,"statfs":{"total":96636764160,"available":96332136448,"internally_reserved":0,"allocated":204800,"data_stored":73385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53200000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297488,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074352,"statfs":{"total":96636764160,"available":96332136448,"internally_reserved":0,"allocated":204800,"data_stored":74184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42699999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297488,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074352,"statfs":{"total":96636764160,"available":96332136448,"internally_reserved":0,"allocated":204800,"data_stored":72921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29399999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297508,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074332,"statfs":{"total":96636764160,"available":96332115968,"internally_reserved":0,"allocated":225280,"data_stored":75776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97899999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297492,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074348,"statfs":{"total":96636764160,"available":96332132352,"internally_reserved":0,"allocated":208896,"data_stored":73455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51300000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38800000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-06T01:41:22.378 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-06T01:41:22.378 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-06T01:41:22.378 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-06T01:41:22.378 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-06T01:41:22.379 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-06T01:41:22.531 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.530+0000 7fded9c22700 1 Processor -- start 2023-10-06T01:41:22.531 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.531+0000 7fded9c22700 1 -- start start 2023-10-06T01:41:22.533 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.531+0000 7fded9c22700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fded415d930 0x7fded40d0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:22.533 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.531+0000 7fded9c22700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded40b2ad0 con 0x7fded41591b0 2023-10-06T01:41:22.533 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.531+0000 7fded9c22700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded40b4160 con 0x7fded415d560 2023-10-06T01:41:22.533 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.531+0000 7fded9c22700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fded40b2230 con 0x7fded415d930 2023-10-06T01:41:22.533 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.531+0000 7fded2ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fded415d930 0x7fded40d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:22.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.532+0000 7fded3fff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fded41591b0 0x7fded4159600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46228/0 (socket says 172.21.15.27:46228) 2023-10-06T01:41:22.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.532+0000 7fded3fff700 1 -- 172.21.15.27:0/197607031 learned_addr learned my addr 172.21.15.27:0/197607031 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:22.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.532+0000 7fded2ffd700 1 -- 172.21.15.27:0/197607031 >> v1:172.21.15.124:6789/0 conn(0x7fded415d560 legacy=0x7fded40b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:22.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.532+0000 7fded2ffd700 1 -- 172.21.15.27:0/197607031 >> v1:172.21.15.27:6789/0 conn(0x7fded41591b0 legacy=0x7fded4159600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:22.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.532+0000 7fded2ffd700 1 -- 172.21.15.27:0/197607031 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded40b2510 con 0x7fded415d930 2023-10-06T01:41:22.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.532+0000 7fded2ffd700 1 --2- 172.21.15.27:0/197607031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fded415d930 0x7fded40d0030 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdec80098e0 tx=0x7fdec80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=431cdd73abab6953 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:22.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.532+0000 7fded27fc700 1 -- 172.21.15.27:0/197607031 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec8009e80 con 0x7fded415d930 2023-10-06T01:41:22.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.532+0000 7fded27fc700 1 -- 172.21.15.27:0/197607031 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdec800b860 con 0x7fded415d930 2023-10-06T01:41:22.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.532+0000 7fded27fc700 1 -- 172.21.15.27:0/197607031 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec800bbe0 con 0x7fded415d930 2023-10-06T01:41:22.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.533+0000 7fded9c22700 1 -- 172.21.15.27:0/197607031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fded415d930 msgr2=0x7fded40d0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:22.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.533+0000 7fded9c22700 1 --2- 172.21.15.27:0/197607031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fded415d930 0x7fded40d0030 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdec80098e0 tx=0x7fdec80091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:22.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.533+0000 7fded9c22700 1 -- 172.21.15.27:0/197607031 shutdown_connections 2023-10-06T01:41:22.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.533+0000 7fded9c22700 1 --2- 172.21.15.27:0/197607031 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fded415d930 0x7fded40d0030 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:22.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.533+0000 7fded9c22700 1 -- 172.21.15.27:0/197607031 >> 172.21.15.27:0/197607031 conn(0x7fded40d1180 msgr2=0x7fded4069640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:22.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.533+0000 7fded9c22700 1 -- 172.21.15.27:0/197607031 shutdown_connections 2023-10-06T01:41:22.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.533+0000 7fded9c22700 1 -- 172.21.15.27:0/197607031 wait complete. 2023-10-06T01:41:22.537 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.534+0000 7fded9c22700 1 Processor -- start 2023-10-06T01:41:22.537 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.534+0000 7fded9c22700 1 -- start start 2023-10-06T01:41:22.537 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.535+0000 7fded9c22700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fded41591b0 0x7fded414b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:22.538 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.535+0000 7fded9c22700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fded415d560 0x7fded40ae990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:22.538 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.535+0000 7fded9c22700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fded415d930 0x7fded40a82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:22.538 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.535+0000 7fded9c22700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded40a7e60 con 0x7fded415d560 2023-10-06T01:41:22.538 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.535+0000 7fded9c22700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded40a6f70 con 0x7fded41591b0 2023-10-06T01:41:22.538 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.535+0000 7fded9c22700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fded40a7250 con 0x7fded415d930 2023-10-06T01:41:22.538 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.535+0000 7fded2ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fded415d560 0x7fded40ae990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:22.538 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.535+0000 7fded2ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fded415d560 0x7fded40ae990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55840/0 (socket says 172.21.15.27:55840) 2023-10-06T01:41:22.538 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.535+0000 7fded2ffd700 1 -- 172.21.15.27:0/3277727968 learned_addr learned my addr 172.21.15.27:0/3277727968 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:22.538 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.535+0000 7fded37fe700 1 --2- 172.21.15.27:0/3277727968 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fded41591b0 0x7fded414b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:22.538 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.535+0000 7fded3fff700 1 --2- 172.21.15.27:0/3277727968 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fded415d930 0x7fded40a82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:22.538 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.535+0000 7fded2ffd700 1 -- 172.21.15.27:0/3277727968 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fded415d930 msgr2=0x7fded40a82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:22.538 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.535+0000 7fded2ffd700 1 --2- 172.21.15.27:0/3277727968 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fded415d930 0x7fded40a82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:22.539 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.535+0000 7fded2ffd700 1 -- 172.21.15.27:0/3277727968 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fded41591b0 msgr2=0x7fded414b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:22.539 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.535+0000 7fded2ffd700 1 --2- 172.21.15.27:0/3277727968 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fded41591b0 0x7fded414b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:22.539 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.535+0000 7fded2ffd700 1 -- 172.21.15.27:0/3277727968 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec8009420 con 0x7fded415d560 2023-10-06T01:41:22.539 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.535+0000 7fded2ffd700 1 --2- 172.21.15.27:0/3277727968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fded415d560 0x7fded40ae990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fded4064a20 tx=0x7fdec8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:22.539 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.535+0000 7fdebb7fe700 1 -- 172.21.15.27:0/3277727968 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec8011420 con 0x7fded415d560 2023-10-06T01:41:22.539 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.535+0000 7fdebb7fe700 1 -- 172.21.15.27:0/3277727968 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdec8011580 con 0x7fded415d560 2023-10-06T01:41:22.539 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.536+0000 7fded9c22700 1 -- 172.21.15.27:0/3277727968 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fded40a7530 con 0x7fded415d560 2023-10-06T01:41:22.539 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.536+0000 7fdebb7fe700 1 -- 172.21.15.27:0/3277727968 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec8011940 con 0x7fded415d560 2023-10-06T01:41:22.539 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.536+0000 7fded9c22700 1 -- 172.21.15.27:0/3277727968 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fded40a7af0 con 0x7fded415d560 2023-10-06T01:41:22.539 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.536+0000 7fded9c22700 1 -- 172.21.15.27:0/3277727968 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fded40a7fd0 con 0x7fded415d560 2023-10-06T01:41:22.539 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.537+0000 7fdebb7fe700 1 -- 172.21.15.27:0/3277727968 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fdec8011cb0 con 0x7fded415d560 2023-10-06T01:41:22.539 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.537+0000 7fdebb7fe700 1 --2- 172.21.15.27:0/3277727968 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdec0066f30 0x7fdec00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:22.540 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.537+0000 7fdebb7fe700 1 -- 172.21.15.27:0/3277727968 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fdec8086030 con 0x7fded415d560 2023-10-06T01:41:22.540 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.538+0000 7fded37fe700 1 --2- 172.21.15.27:0/3277727968 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdec0066f30 0x7fdec00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:22.540 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.538+0000 7fded37fe700 1 --2- 172.21.15.27:0/3277727968 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdec0066f30 0x7fdec00693e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdec4003860 tx=0x7fdec4009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:22.541 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.541+0000 7fdebb7fe700 1 -- 172.21.15.27:0/3277727968 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdec8053140 con 0x7fded415d560 2023-10-06T01:41:22.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.790+0000 7fded9c22700 1 -- 172.21.15.27:0/3277727968 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fded4064050 con 0x7fded415d560 2023-10-06T01:41:22.793 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.793+0000 7fdebb7fe700 1 -- 172.21.15.27:0/3277727968 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+163706 (secure 0 0 0) 0x7fdec8056760 con 0x7fded415d560 2023-10-06T01:41:22.795 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:22.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.795+0000 7fded9c22700 1 -- 172.21.15.27:0/3277727968 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdec0066f30 msgr2=0x7fdec00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:22.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.795+0000 7fded9c22700 1 --2- 172.21.15.27:0/3277727968 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdec0066f30 0x7fdec00693e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fdec4003860 tx=0x7fdec4009040 comp rx=0 tx=0).stop 2023-10-06T01:41:22.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.795+0000 7fded9c22700 1 -- 172.21.15.27:0/3277727968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fded415d560 msgr2=0x7fded40ae990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:22.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.795+0000 7fded9c22700 1 --2- 172.21.15.27:0/3277727968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fded415d560 0x7fded40ae990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fded4064a20 tx=0x7fdec8017000 comp rx=0 tx=0).stop 2023-10-06T01:41:22.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.795+0000 7fded9c22700 1 -- 172.21.15.27:0/3277727968 shutdown_connections 2023-10-06T01:41:22.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.795+0000 7fded9c22700 1 --2- 172.21.15.27:0/3277727968 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdec0066f30 0x7fdec00693e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:22.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.795+0000 7fded9c22700 1 --2- 172.21.15.27:0/3277727968 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fded415d930 0x7fded40a82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:22.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.795+0000 7fded9c22700 1 --2- 172.21.15.27:0/3277727968 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fded41591b0 0x7fded414b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:22.798 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.795+0000 7fded9c22700 1 --2- 172.21.15.27:0/3277727968 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fded415d560 0x7fded40ae990 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:22.798 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.795+0000 7fded9c22700 1 -- 172.21.15.27:0/3277727968 >> 172.21.15.27:0/3277727968 conn(0x7fded40d1180 msgr2=0x7fded40682c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:22.798 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.795+0000 7fded9c22700 1 -- 172.21.15.27:0/3277727968 shutdown_connections 2023-10-06T01:41:22.798 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.795+0000 7fded9c22700 1 -- 172.21.15.27:0/3277727968 wait complete. 2023-10-06T01:41:22.808 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":5,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6832","nonce":90278},{"type":"v1","addr":"172.21.15.124:6833","nonce":90278}]},"active_addr":"172.21.15.124:6833/90278","active_change":"2023-10-06T01:41:00.027960+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":4107,"name":"y","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.124:0","nonce":3949624554}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.124:0","nonce":2846733293}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.124:0","nonce":2608355337}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.124:0","nonce":3215948734}]}]} 2023-10-06T01:41:22.812 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-06T01:41:22.812 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-06T01:41:22.812 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T01:41:22.938 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.930+0000 7f48c233f700 1 Processor -- start 2023-10-06T01:41:22.939 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.930+0000 7f48c233f700 1 -- start start 2023-10-06T01:41:22.939 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.930+0000 7f48c233f700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f48bc159580 0x7f48bc0b2790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:22.939 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.930+0000 7f48c233f700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48bc0ad3c0 con 0x7f48bc159950 2023-10-06T01:41:22.939 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.930+0000 7f48c233f700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48bc0b0ec0 con 0x7f48bc0ae860 2023-10-06T01:41:22.939 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.930+0000 7f48c233f700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48bc0b11a0 con 0x7f48bc159580 2023-10-06T01:41:22.939 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.931+0000 7f48bb7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f48bc159950 0x7f48bc0b0190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46236/0 (socket says 172.21.15.27:46236) 2023-10-06T01:41:22.939 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.931+0000 7f48bb7fe700 1 -- 172.21.15.27:0/686701757 learned_addr learned my addr 172.21.15.27:0/686701757 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:22.939 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.931+0000 7f48bbfff700 1 --2- 172.21.15.27:0/686701757 >> v2:172.21.15.124:3300/0 conn(0x7f48bc159580 0x7f48bc0b2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:22.939 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.931+0000 7f48baffd700 1 -- 172.21.15.27:0/686701757 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 803886047 0 0) 0x7f48bc0ad3c0 con 0x7f48bc159950 2023-10-06T01:41:22.939 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.931+0000 7f48baffd700 1 -- 172.21.15.27:0/686701757 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48a4003570 con 0x7f48bc159950 2023-10-06T01:41:22.939 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.931+0000 7f48baffd700 1 -- 172.21.15.27:0/686701757 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2275517692 0 0) 0x7f48a4003570 con 0x7f48bc159950 2023-10-06T01:41:22.940 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.931+0000 7f48baffd700 1 -- 172.21.15.27:0/686701757 >> v2:172.21.15.124:3300/0 conn(0x7f48bc159580 msgr2=0x7f48bc0b2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:22.940 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.931+0000 7f48baffd700 1 --2- 172.21.15.27:0/686701757 >> v2:172.21.15.124:3300/0 conn(0x7f48bc159580 0x7f48bc0b2790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:22.940 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.931+0000 7f48baffd700 1 -- 172.21.15.27:0/686701757 >> v1:172.21.15.124:6789/0 conn(0x7f48bc0ae860 legacy=0x7f48bc163330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:22.940 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.931+0000 7f48baffd700 1 -- 172.21.15.27:0/686701757 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48bc0acd70 con 0x7f48bc159950 2023-10-06T01:41:22.940 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.931+0000 7f48baffd700 1 -- 172.21.15.27:0/686701757 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2619078944 0 0) 0x7f48ac001470 con 0x7f48bc159950 2023-10-06T01:41:22.940 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.931+0000 7f48baffd700 1 -- 172.21.15.27:0/686701757 >> v1:172.21.15.27:6789/0 conn(0x7f48bc159950 legacy=0x7f48bc0b0190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:22.940 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.931+0000 7f48baffd700 1 --2- 172.21.15.27:0/686701757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48a4003a60 0x7f48a4003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:22.940 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.931+0000 7f48baffd700 1 --2- 172.21.15.27:0/686701757 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f48a4004810 0x7f48a4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:22.940 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.931+0000 7f48baffd700 1 --2- 172.21.15.27:0/686701757 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f48a40051c0 0x7f48a400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:22.940 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.931+0000 7f48baffd700 1 -- 172.21.15.27:0/686701757 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48bc0b11a0 con 0x7f48a4003a60 2023-10-06T01:41:22.940 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.931+0000 7f48baffd700 1 -- 172.21.15.27:0/686701757 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a400c0d0 con 0x7f48a40051c0 2023-10-06T01:41:22.940 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.931+0000 7f48baffd700 1 -- 172.21.15.27:0/686701757 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48a400c290 con 0x7f48a4004810 2023-10-06T01:41:22.941 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.931+0000 7f48bbfff700 1 --2- 172.21.15.27:0/686701757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48a4003a60 0x7f48a4003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:22.941 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.931+0000 7f48bb7fe700 1 --2- 172.21.15.27:0/686701757 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f48a4004810 0x7f48a4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:22.941 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.931+0000 7f48c08dc700 1 --2- 172.21.15.27:0/686701757 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f48a40051c0 0x7f48a400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:22.941 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.932+0000 7f48bbfff700 1 -- 172.21.15.27:0/686701757 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f48a4004810 msgr2=0x7f48a4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:22.941 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.932+0000 7f48bbfff700 1 --2- 172.21.15.27:0/686701757 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f48a4004810 0x7f48a4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:22.941 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.932+0000 7f48bbfff700 1 -- 172.21.15.27:0/686701757 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f48a40051c0 msgr2=0x7f48a400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:22.941 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.932+0000 7f48bbfff700 1 --2- 172.21.15.27:0/686701757 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f48a40051c0 0x7f48a400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:22.942 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.932+0000 7f48c08dc700 1 -- 172.21.15.27:0/686701757 reap_dead start 2023-10-06T01:41:22.942 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.932+0000 7f48bbfff700 1 -- 172.21.15.27:0/686701757 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a4003570 con 0x7f48a4003a60 2023-10-06T01:41:22.942 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.932+0000 7f48bb7fe700 1 --2- 172.21.15.27:0/686701757 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f48a4004810 0x7f48a4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:22.942 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.932+0000 7f48bbfff700 1 --2- 172.21.15.27:0/686701757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48a4003a60 0x7f48a4003e70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f48b0009c00 tx=0x7f48b00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47c4fd8e8cb2455 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:22.942 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.932+0000 7f48baffd700 1 -- 172.21.15.27:0/686701757 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b000e710 con 0x7f48a4003a60 2023-10-06T01:41:22.942 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.932+0000 7f48baffd700 1 -- 172.21.15.27:0/686701757 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48b000e870 con 0x7f48a4003a60 2023-10-06T01:41:22.942 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.932+0000 7f48baffd700 1 -- 172.21.15.27:0/686701757 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48b000ec30 con 0x7f48a4003a60 2023-10-06T01:41:22.942 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.932+0000 7f48c233f700 1 -- 172.21.15.27:0/686701757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48a4003a60 msgr2=0x7f48a4003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:22.943 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.932+0000 7f48c233f700 1 --2- 172.21.15.27:0/686701757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48a4003a60 0x7f48a4003e70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f48b0009c00 tx=0x7f48b00091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:22.943 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.932+0000 7f48c233f700 1 -- 172.21.15.27:0/686701757 shutdown_connections 2023-10-06T01:41:22.943 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.932+0000 7f48c233f700 1 --2- 172.21.15.27:0/686701757 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48a4003a60 0x7f48a4003e70 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:22.943 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.932+0000 7f48c233f700 1 -- 172.21.15.27:0/686701757 >> 172.21.15.27:0/686701757 conn(0x7f48bc06e6f0 msgr2=0x7f48bc06afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:22.943 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.932+0000 7f48c233f700 1 -- 172.21.15.27:0/686701757 shutdown_connections 2023-10-06T01:41:22.943 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.932+0000 7f48c233f700 1 -- 172.21.15.27:0/686701757 wait complete. 2023-10-06T01:41:22.943 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.933+0000 7f48c233f700 1 Processor -- start 2023-10-06T01:41:22.943 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.934+0000 7f48c233f700 1 -- start start 2023-10-06T01:41:22.943 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.934+0000 7f48c233f700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f48a4003a60 0x7f48bc14b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:22.943 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.934+0000 7f48c233f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48bc14ad90 0x7f48bc14d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:22.943 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.934+0000 7f48c233f700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f48bc14dca0 0x7f48bc0aa040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:22.944 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.934+0000 7f48c233f700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48bc14bfd0 con 0x7f48bc14ad90 2023-10-06T01:41:22.944 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.934+0000 7f48c233f700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48bc14e1d0 con 0x7f48bc14dca0 2023-10-06T01:41:22.944 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.934+0000 7f48c233f700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48bc14e490 con 0x7f48a4003a60 2023-10-06T01:41:22.944 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.934+0000 7f48bb7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48bc14ad90 0x7f48bc14d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:22.944 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.934+0000 7f48bb7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48bc14ad90 0x7f48bc14d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55862/0 (socket says 172.21.15.27:55862) 2023-10-06T01:41:22.944 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.934+0000 7f48bb7fe700 1 -- 172.21.15.27:0/1276196043 learned_addr learned my addr 172.21.15.27:0/1276196043 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:22.944 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.935+0000 7f48c08dc700 1 --2- 172.21.15.27:0/1276196043 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f48bc14dca0 0x7f48bc0aa040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:22.944 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.935+0000 7f48bbfff700 1 --2- 172.21.15.27:0/1276196043 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f48a4003a60 0x7f48bc14b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:22.944 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.935+0000 7f48bb7fe700 1 -- 172.21.15.27:0/1276196043 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f48a4003a60 msgr2=0x7f48bc14b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:22.944 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.935+0000 7f48bb7fe700 1 --2- 172.21.15.27:0/1276196043 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f48a4003a60 0x7f48bc14b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:22.944 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.935+0000 7f48bb7fe700 1 -- 172.21.15.27:0/1276196043 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f48bc14dca0 msgr2=0x7f48bc0aa040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:22.945 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.935+0000 7f48bb7fe700 1 --2- 172.21.15.27:0/1276196043 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f48bc14dca0 0x7f48bc0aa040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:22.945 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.935+0000 7f48bb7fe700 1 -- 172.21.15.27:0/1276196043 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48bc14e770 con 0x7f48bc14ad90 2023-10-06T01:41:22.945 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.935+0000 7f48bb7fe700 1 --2- 172.21.15.27:0/1276196043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48bc14ad90 0x7f48bc14d760 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f48ac001200 tx=0x7f48ac001e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:22.945 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.935+0000 7f48a3fff700 1 -- 172.21.15.27:0/1276196043 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ac00c7b0 con 0x7f48bc14ad90 2023-10-06T01:41:22.945 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.935+0000 7f48a3fff700 1 -- 172.21.15.27:0/1276196043 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48ac00c910 con 0x7f48bc14ad90 2023-10-06T01:41:22.945 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.935+0000 7f48c233f700 1 -- 172.21.15.27:0/1276196043 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48bc14ea50 con 0x7f48bc14ad90 2023-10-06T01:41:22.945 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.935+0000 7f48a3fff700 1 -- 172.21.15.27:0/1276196043 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48ac00ccb0 con 0x7f48bc14ad90 2023-10-06T01:41:22.945 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.935+0000 7f48c233f700 1 -- 172.21.15.27:0/1276196043 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48bc0aa580 con 0x7f48bc14ad90 2023-10-06T01:41:22.945 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.936+0000 7f48c233f700 1 -- 172.21.15.27:0/1276196043 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48bc15a210 con 0x7f48bc14ad90 2023-10-06T01:41:22.945 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.936+0000 7f48a3fff700 1 -- 172.21.15.27:0/1276196043 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f48ac013630 con 0x7f48bc14ad90 2023-10-06T01:41:22.945 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.937+0000 7f48a3fff700 1 --2- 172.21.15.27:0/1276196043 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f48a8066f30 0x7f48a80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:22.946 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.937+0000 7f48a3fff700 1 -- 172.21.15.27:0/1276196043 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f48ac084690 con 0x7f48bc14ad90 2023-10-06T01:41:22.946 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.937+0000 7f48bbfff700 1 --2- 172.21.15.27:0/1276196043 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f48a8066f30 0x7f48a80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:22.946 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.937+0000 7f48bbfff700 1 --2- 172.21.15.27:0/1276196043 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f48a8066f30 0x7f48a80693e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f48b00091e0 tx=0x7f48b001a000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:22.946 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:22.941+0000 7f48a3fff700 1 -- 172.21.15.27:0/1276196043 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f48ac0517a0 con 0x7f48bc14ad90 2023-10-06T01:41:23.136 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.136+0000 7f48c233f700 1 -- 172.21.15.27:0/1276196043 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48bc064050 con 0x7f48bc14ad90 2023-10-06T01:41:23.137 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.137+0000 7f48a3fff700 1 -- 172.21.15.27:0/1276196043 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16866 (secure 0 0 0) 0x7f48ac054dc0 con 0x7f48bc14ad90 2023-10-06T01:41:23.137 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:23.137 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":20,"fsid":"20a7bff0-74f8-4f2f-b6c6-899e804bc965","created":"2023-10-06T01:40:54.474215+0000","modified":"2023-10-06T01:41:14.324546+0000","last_up_change":"2023-10-06T01:41:05.242847+0000","last_in_change":"2023-10-06T01:40:56.890960+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-10-06T01:41:05.685386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-10-06T01:41:06.092995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T01:41:10.434257+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T01:41:11.700262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b902c86d-800c-40c5-b152-b545e50d02c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":90644},{"type":"v1","addr":"172.21.15.27:6801","nonce":90644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":90644},{"type":"v1","addr":"172.21.15.27:6803","nonce":90644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":90644},{"type":"v1","addr":"172.21.15.27:6807","nonce":90644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":90644},{"type":"v1","addr":"172.21.15.27:6805","nonce":90644}]},"public_addr":"172.21.15.27:6801/90644","cluster_addr":"172.21.15.27:6803/90644","heartbeat_back_addr":"172.21.15.27:6807/90644","heartbeat_front_addr":"172.21.15.27:6805/90644","state":["exists","up"]},{"osd":1,"uuid":"5b4a70e5-b06e-48cd-bfc4-7962c5d05386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":90643},{"type":"v1","addr":"172.21.15.27:6809","nonce":90643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":90643},{"type":"v1","addr":"172.21.15.27:6811","nonce":90643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":90643},{"type":"v1","addr":"172.21.15.27:6815","nonce":90643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":90643},{"type":"v1","addr":"172.21.15.27:6813","nonce":90643}]},"public_addr":"172.21.15.27:6809/90643","cluster_addr":"172.21.15.27:6811/90643","heartbeat_back_addr":"172.21.15.27:6815/90643","heartbeat_front_addr":"172.21.15.27:6813/90643","state":["exists","up"]},{"osd":2,"uuid":"22d9de46-4073-4336-9122-5227e601eca0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":90645},{"type":"v1","addr":"172.21.15.27:6817","nonce":90645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":90645},{"type":"v1","addr":"172.21.15.27:6819","nonce":90645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":90645},{"type":"v1","addr":"172.21.15.27:6823","nonce":90645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":90645},{"type":"v1","addr":"172.21.15.27:6821","nonce":90645}]},"public_addr":"172.21.15.27:6817/90645","cluster_addr":"172.21.15.27:6819/90645","heartbeat_back_addr":"172.21.15.27:6823/90645","heartbeat_front_addr":"172.21.15.27:6821/90645","state":["exists","up"]},{"osd":3,"uuid":"438bb2d3-34e2-4b7c-9734-8360c1b67650","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":90668},{"type":"v1","addr":"172.21.15.27:6825","nonce":90668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":90668},{"type":"v1","addr":"172.21.15.27:6827","nonce":90668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":90668},{"type":"v1","addr":"172.21.15.27:6831","nonce":90668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":90668},{"type":"v1","addr":"172.21.15.27:6829","nonce":90668}]},"public_addr":"172.21.15.27:6825/90668","cluster_addr":"172.21.15.27:6827/90668","heartbeat_back_addr":"172.21.15.27:6831/90668","heartbeat_front_addr":"172.21.15.27:6829/90668","state":["exists","up"]},{"osd":4,"uuid":"e1956a8e-e0f8-4e7c-acbb-40037415a354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":90863},{"type":"v1","addr":"172.21.15.124:6801","nonce":90863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":90863},{"type":"v1","addr":"172.21.15.124:6803","nonce":90863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":90863},{"type":"v1","addr":"172.21.15.124:6807","nonce":90863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":90863},{"type":"v1","addr":"172.21.15.124:6805","nonce":90863}]},"public_addr":"172.21.15.124:6801/90863","cluster_addr":"172.21.15.124:6803/90863","heartbeat_back_addr":"172.21.15.124:6807/90863","heartbeat_front_addr":"172.21.15.124:6805/90863","state":["exists","up"]},{"osd":5,"uuid":"306c59d4-846d-4b73-b1c9-051bec9e9368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":90885},{"type":"v1","addr":"172.21.15.124:6817","nonce":90885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":90885},{"type":"v1","addr":"172.21.15.124:6819","nonce":90885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":90885},{"type":"v1","addr":"172.21.15.124:6823","nonce":90885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":90885},{"type":"v1","addr":"172.21.15.124:6821","nonce":90885}]},"public_addr":"172.21.15.124:6817/90885","cluster_addr":"172.21.15.124:6819/90885","heartbeat_back_addr":"172.21.15.124:6823/90885","heartbeat_front_addr":"172.21.15.124:6821/90885","state":["exists","up"]},{"osd":6,"uuid":"15463e53-ac8d-47d1-b4f6-c0d31f29e4a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":90864},{"type":"v1","addr":"172.21.15.124:6809","nonce":90864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":90864},{"type":"v1","addr":"172.21.15.124:6811","nonce":90864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":90864},{"type":"v1","addr":"172.21.15.124:6815","nonce":90864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":90864},{"type":"v1","addr":"172.21.15.124:6813","nonce":90864}]},"public_addr":"172.21.15.124:6809/90864","cluster_addr":"172.21.15.124:6811/90864","heartbeat_back_addr":"172.21.15.124:6815/90864","heartbeat_front_addr":"172.21.15.124:6813/90864","state":["exists","up"]},{"osd":7,"uuid":"ae463a90-a9d3-498f-86fa-47a35b3fe889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":90894},{"type":"v1","addr":"172.21.15.124:6825","nonce":90894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":90894},{"type":"v1","addr":"172.21.15.124:6827","nonce":90894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":90894},{"type":"v1","addr":"172.21.15.124:6831","nonce":90894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":90894},{"type":"v1","addr":"172.21.15.124:6829","nonce":90894}]},"public_addr":"172.21.15.124:6825/90894","cluster_addr":"172.21.15.124:6827/90894","heartbeat_back_addr":"172.21.15.124:6831/90894","heartbeat_front_addr":"172.21.15.124:6829/90894","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.827704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.839951+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.032230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.050605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.987908+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.031290+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.080707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.053692+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T01:41:23.139 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.139+0000 7f48c233f700 1 -- 172.21.15.27:0/1276196043 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f48a8066f30 msgr2=0x7f48a80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.139 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.139+0000 7f48c233f700 1 --2- 172.21.15.27:0/1276196043 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f48a8066f30 0x7f48a80693e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f48b00091e0 tx=0x7f48b001a000 comp rx=0 tx=0).stop 2023-10-06T01:41:23.140 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.139+0000 7f48c233f700 1 -- 172.21.15.27:0/1276196043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48bc14ad90 msgr2=0x7f48bc14d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.140 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.139+0000 7f48c233f700 1 --2- 172.21.15.27:0/1276196043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48bc14ad90 0x7f48bc14d760 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f48ac001200 tx=0x7f48ac001e20 comp rx=0 tx=0).stop 2023-10-06T01:41:23.140 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.139+0000 7f48c233f700 1 -- 172.21.15.27:0/1276196043 shutdown_connections 2023-10-06T01:41:23.140 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.139+0000 7f48c233f700 1 --2- 172.21.15.27:0/1276196043 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f48a8066f30 0x7f48a80693e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.140 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.139+0000 7f48c233f700 1 --2- 172.21.15.27:0/1276196043 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f48a4003a60 0x7f48bc14b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.140 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.139+0000 7f48c233f700 1 --2- 172.21.15.27:0/1276196043 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f48bc14dca0 0x7f48bc0aa040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.141 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.139+0000 7f48c233f700 1 --2- 172.21.15.27:0/1276196043 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f48bc14ad90 0x7f48bc14d760 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.141 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.139+0000 7f48c233f700 1 -- 172.21.15.27:0/1276196043 >> 172.21.15.27:0/1276196043 conn(0x7f48bc06e6f0 msgr2=0x7f48bc15e920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:23.141 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.139+0000 7f48c233f700 1 -- 172.21.15.27:0/1276196043 shutdown_connections 2023-10-06T01:41:23.141 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.139+0000 7f48c233f700 1 -- 172.21.15.27:0/1276196043 wait complete. 2023-10-06T01:41:23.151 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-06T01:41:23.152 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T01:41:23.271 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.270+0000 7f385cf33700 1 Processor -- start 2023-10-06T01:41:23.271 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.270+0000 7f385cf33700 1 -- start start 2023-10-06T01:41:23.278 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.271+0000 7f385cf33700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f385815d560 0x7f38580d0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.278 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.271+0000 7f385cf33700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f38581591b0 0x7f3858159630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.278 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.271+0000 7f385cf33700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38580b2ad0 con 0x7f385815d930 2023-10-06T01:41:23.279 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.271+0000 7f385cf33700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38580b4160 con 0x7f38581591b0 2023-10-06T01:41:23.279 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.271+0000 7f385cf33700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3858159b70 con 0x7f385815d560 2023-10-06T01:41:23.279 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.271+0000 7f3855d9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f385815d930 0x7f38580b1a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46246/0 (socket says 172.21.15.27:46246) 2023-10-06T01:41:23.279 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.271+0000 7f3855d9b700 1 -- 172.21.15.27:0/2686636399 learned_addr learned my addr 172.21.15.27:0/2686636399 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:23.279 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.271+0000 7f3856d9d700 1 --2- 172.21.15.27:0/2686636399 >> v2:172.21.15.124:3300/0 conn(0x7f38581591b0 0x7f3858159630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.279 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.271+0000 7f385659c700 1 --2- 172.21.15.27:0/2686636399 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f385815d560 0x7f38580d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.280 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.271+0000 7f385559a700 1 -- 172.21.15.27:0/2686636399 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 31900562 0 0) 0x7f38580b2ad0 con 0x7f385815d930 2023-10-06T01:41:23.280 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.272+0000 7f385559a700 1 -- 172.21.15.27:0/2686636399 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3844003570 con 0x7f385815d930 2023-10-06T01:41:23.280 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.272+0000 7f3856d9d700 1 -- 172.21.15.27:0/2686636399 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f385815d560 msgr2=0x7f38580d0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:23.280 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.272+0000 7f3856d9d700 1 --2- 172.21.15.27:0/2686636399 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f385815d560 0x7f38580d0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.280 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.272+0000 7f3856d9d700 1 -- 172.21.15.27:0/2686636399 >> v1:172.21.15.27:6789/0 conn(0x7f385815d930 legacy=0x7f38580b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.280 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.272+0000 7f3856d9d700 1 -- 172.21.15.27:0/2686636399 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38580b2230 con 0x7f38581591b0 2023-10-06T01:41:23.280 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.272+0000 7f3856d9d700 1 --2- 172.21.15.27:0/2686636399 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f38581591b0 0x7f3858159630 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3848009e60 tx=0x7f3848009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=efd04526e59aba66 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:23.281 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.272+0000 7f385559a700 1 -- 172.21.15.27:0/2686636399 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f384800e070 con 0x7f38581591b0 2023-10-06T01:41:23.281 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.272+0000 7f385559a700 1 -- 172.21.15.27:0/2686636399 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f384800bab0 con 0x7f38581591b0 2023-10-06T01:41:23.281 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.272+0000 7f385559a700 1 -- 172.21.15.27:0/2686636399 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f384800be70 con 0x7f38581591b0 2023-10-06T01:41:23.281 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.273+0000 7f385cf33700 1 -- 172.21.15.27:0/2686636399 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f38581591b0 msgr2=0x7f3858159630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.281 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.273+0000 7f385cf33700 1 --2- 172.21.15.27:0/2686636399 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f38581591b0 0x7f3858159630 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3848009e60 tx=0x7f3848009780 comp rx=0 tx=0).stop 2023-10-06T01:41:23.281 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.273+0000 7f385cf33700 1 -- 172.21.15.27:0/2686636399 shutdown_connections 2023-10-06T01:41:23.282 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.273+0000 7f385cf33700 1 --2- 172.21.15.27:0/2686636399 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f38581591b0 0x7f3858159630 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.282 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.273+0000 7f385cf33700 1 --2- 172.21.15.27:0/2686636399 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f385815d560 0x7f38580d0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.282 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.273+0000 7f385cf33700 1 -- 172.21.15.27:0/2686636399 >> 172.21.15.27:0/2686636399 conn(0x7f38580d1180 msgr2=0x7f3858069640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:23.282 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.273+0000 7f385cf33700 1 -- 172.21.15.27:0/2686636399 shutdown_connections 2023-10-06T01:41:23.282 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.273+0000 7f385cf33700 1 -- 172.21.15.27:0/2686636399 wait complete. 2023-10-06T01:41:23.282 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.274+0000 7f385cf33700 1 Processor -- start 2023-10-06T01:41:23.283 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.274+0000 7f385cf33700 1 -- start start 2023-10-06T01:41:23.283 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.275+0000 7f385cf33700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38581591b0 0x7f385815b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.283 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.275+0000 7f385cf33700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f385815d560 0x7f3858145020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.283 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.275+0000 7f385cf33700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f385815d930 0x7f3858154780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.283 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.275+0000 7f385cf33700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3858152710 con 0x7f38581591b0 2023-10-06T01:41:23.283 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.275+0000 7f385cf33700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f385815ac60 con 0x7f385815d930 2023-10-06T01:41:23.283 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.275+0000 7f385cf33700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f385815a050 con 0x7f385815d560 2023-10-06T01:41:23.284 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.275+0000 7f385659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38581591b0 0x7f385815b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.284 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.275+0000 7f385659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38581591b0 0x7f385815b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55866/0 (socket says 172.21.15.27:55866) 2023-10-06T01:41:23.284 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.275+0000 7f385659c700 1 -- 172.21.15.27:0/3431811138 learned_addr learned my addr 172.21.15.27:0/3431811138 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:23.284 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.275+0000 7f3855d9b700 1 --2- 172.21.15.27:0/3431811138 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f385815d560 0x7f3858145020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.284 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.275+0000 7f3856d9d700 1 --2- 172.21.15.27:0/3431811138 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f385815d930 0x7f3858154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.284 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.275+0000 7f385659c700 1 -- 172.21.15.27:0/3431811138 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f385815d560 msgr2=0x7f3858145020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.285 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.275+0000 7f385659c700 1 --2- 172.21.15.27:0/3431811138 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f385815d560 0x7f3858145020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.285 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.275+0000 7f385659c700 1 -- 172.21.15.27:0/3431811138 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f385815d930 msgr2=0x7f3858154780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.285 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.275+0000 7f385659c700 1 --2- 172.21.15.27:0/3431811138 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f385815d930 0x7f3858154780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.285 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.275+0000 7f385659c700 1 -- 172.21.15.27:0/3431811138 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3848009a20 con 0x7f38581591b0 2023-10-06T01:41:23.285 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.275+0000 7f385659c700 1 --2- 172.21.15.27:0/3431811138 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38581591b0 0x7f385815b0f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f385000ec80 tx=0x7f38500091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:23.285 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.275+0000 7f384e7fc700 1 -- 172.21.15.27:0/3431811138 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f385000c9d0 con 0x7f38581591b0 2023-10-06T01:41:23.286 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.275+0000 7f384e7fc700 1 -- 172.21.15.27:0/3431811138 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f385000cb30 con 0x7f38581591b0 2023-10-06T01:41:23.286 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.275+0000 7f385cf33700 1 -- 172.21.15.27:0/3431811138 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f385815a330 con 0x7f38581591b0 2023-10-06T01:41:23.286 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.275+0000 7f384e7fc700 1 -- 172.21.15.27:0/3431811138 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3850013470 con 0x7f38581591b0 2023-10-06T01:41:23.286 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.275+0000 7f385cf33700 1 -- 172.21.15.27:0/3431811138 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f385815a8f0 con 0x7f38581591b0 2023-10-06T01:41:23.286 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.276+0000 7f385cf33700 1 -- 172.21.15.27:0/3431811138 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3858064050 con 0x7f38581591b0 2023-10-06T01:41:23.286 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.277+0000 7f384e7fc700 1 -- 172.21.15.27:0/3431811138 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3850013820 con 0x7f38581591b0 2023-10-06T01:41:23.286 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.277+0000 7f384e7fc700 1 --2- 172.21.15.27:0/3431811138 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3838066f30 0x7f38380693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.287 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.277+0000 7f384e7fc700 1 -- 172.21.15.27:0/3431811138 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f3850084b20 con 0x7f38581591b0 2023-10-06T01:41:23.287 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.277+0000 7f3855d9b700 1 --2- 172.21.15.27:0/3431811138 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3838066f30 0x7f38380693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.287 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.278+0000 7f3855d9b700 1 --2- 172.21.15.27:0/3431811138 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3838066f30 0x7f38380693e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f38400058d0 tx=0x7f3840009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:23.287 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.282+0000 7f384e7fc700 1 -- 172.21.15.27:0/3431811138 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3850051b90 con 0x7f38581591b0 2023-10-06T01:41:23.478 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.477+0000 7f385cf33700 1 -- 172.21.15.27:0/3431811138 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f38580b2510 con 0x7f38581591b0 2023-10-06T01:41:23.478 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.478+0000 7f384e7fc700 1 -- 172.21.15.27:0/3431811138 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16866 (secure 0 0 0) 0x7f38500551b0 con 0x7f38581591b0 2023-10-06T01:41:23.479 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:23.479 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":20,"fsid":"20a7bff0-74f8-4f2f-b6c6-899e804bc965","created":"2023-10-06T01:40:54.474215+0000","modified":"2023-10-06T01:41:14.324546+0000","last_up_change":"2023-10-06T01:41:05.242847+0000","last_in_change":"2023-10-06T01:40:56.890960+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-10-06T01:41:05.685386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-10-06T01:41:06.092995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T01:41:10.434257+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T01:41:11.700262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b902c86d-800c-40c5-b152-b545e50d02c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":90644},{"type":"v1","addr":"172.21.15.27:6801","nonce":90644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":90644},{"type":"v1","addr":"172.21.15.27:6803","nonce":90644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":90644},{"type":"v1","addr":"172.21.15.27:6807","nonce":90644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":90644},{"type":"v1","addr":"172.21.15.27:6805","nonce":90644}]},"public_addr":"172.21.15.27:6801/90644","cluster_addr":"172.21.15.27:6803/90644","heartbeat_back_addr":"172.21.15.27:6807/90644","heartbeat_front_addr":"172.21.15.27:6805/90644","state":["exists","up"]},{"osd":1,"uuid":"5b4a70e5-b06e-48cd-bfc4-7962c5d05386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":90643},{"type":"v1","addr":"172.21.15.27:6809","nonce":90643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":90643},{"type":"v1","addr":"172.21.15.27:6811","nonce":90643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":90643},{"type":"v1","addr":"172.21.15.27:6815","nonce":90643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":90643},{"type":"v1","addr":"172.21.15.27:6813","nonce":90643}]},"public_addr":"172.21.15.27:6809/90643","cluster_addr":"172.21.15.27:6811/90643","heartbeat_back_addr":"172.21.15.27:6815/90643","heartbeat_front_addr":"172.21.15.27:6813/90643","state":["exists","up"]},{"osd":2,"uuid":"22d9de46-4073-4336-9122-5227e601eca0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":90645},{"type":"v1","addr":"172.21.15.27:6817","nonce":90645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":90645},{"type":"v1","addr":"172.21.15.27:6819","nonce":90645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":90645},{"type":"v1","addr":"172.21.15.27:6823","nonce":90645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":90645},{"type":"v1","addr":"172.21.15.27:6821","nonce":90645}]},"public_addr":"172.21.15.27:6817/90645","cluster_addr":"172.21.15.27:6819/90645","heartbeat_back_addr":"172.21.15.27:6823/90645","heartbeat_front_addr":"172.21.15.27:6821/90645","state":["exists","up"]},{"osd":3,"uuid":"438bb2d3-34e2-4b7c-9734-8360c1b67650","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":90668},{"type":"v1","addr":"172.21.15.27:6825","nonce":90668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":90668},{"type":"v1","addr":"172.21.15.27:6827","nonce":90668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":90668},{"type":"v1","addr":"172.21.15.27:6831","nonce":90668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":90668},{"type":"v1","addr":"172.21.15.27:6829","nonce":90668}]},"public_addr":"172.21.15.27:6825/90668","cluster_addr":"172.21.15.27:6827/90668","heartbeat_back_addr":"172.21.15.27:6831/90668","heartbeat_front_addr":"172.21.15.27:6829/90668","state":["exists","up"]},{"osd":4,"uuid":"e1956a8e-e0f8-4e7c-acbb-40037415a354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":90863},{"type":"v1","addr":"172.21.15.124:6801","nonce":90863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":90863},{"type":"v1","addr":"172.21.15.124:6803","nonce":90863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":90863},{"type":"v1","addr":"172.21.15.124:6807","nonce":90863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":90863},{"type":"v1","addr":"172.21.15.124:6805","nonce":90863}]},"public_addr":"172.21.15.124:6801/90863","cluster_addr":"172.21.15.124:6803/90863","heartbeat_back_addr":"172.21.15.124:6807/90863","heartbeat_front_addr":"172.21.15.124:6805/90863","state":["exists","up"]},{"osd":5,"uuid":"306c59d4-846d-4b73-b1c9-051bec9e9368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":90885},{"type":"v1","addr":"172.21.15.124:6817","nonce":90885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":90885},{"type":"v1","addr":"172.21.15.124:6819","nonce":90885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":90885},{"type":"v1","addr":"172.21.15.124:6823","nonce":90885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":90885},{"type":"v1","addr":"172.21.15.124:6821","nonce":90885}]},"public_addr":"172.21.15.124:6817/90885","cluster_addr":"172.21.15.124:6819/90885","heartbeat_back_addr":"172.21.15.124:6823/90885","heartbeat_front_addr":"172.21.15.124:6821/90885","state":["exists","up"]},{"osd":6,"uuid":"15463e53-ac8d-47d1-b4f6-c0d31f29e4a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":90864},{"type":"v1","addr":"172.21.15.124:6809","nonce":90864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":90864},{"type":"v1","addr":"172.21.15.124:6811","nonce":90864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":90864},{"type":"v1","addr":"172.21.15.124:6815","nonce":90864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":90864},{"type":"v1","addr":"172.21.15.124:6813","nonce":90864}]},"public_addr":"172.21.15.124:6809/90864","cluster_addr":"172.21.15.124:6811/90864","heartbeat_back_addr":"172.21.15.124:6815/90864","heartbeat_front_addr":"172.21.15.124:6813/90864","state":["exists","up"]},{"osd":7,"uuid":"ae463a90-a9d3-498f-86fa-47a35b3fe889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":90894},{"type":"v1","addr":"172.21.15.124:6825","nonce":90894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":90894},{"type":"v1","addr":"172.21.15.124:6827","nonce":90894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":90894},{"type":"v1","addr":"172.21.15.124:6831","nonce":90894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":90894},{"type":"v1","addr":"172.21.15.124:6829","nonce":90894}]},"public_addr":"172.21.15.124:6825/90894","cluster_addr":"172.21.15.124:6827/90894","heartbeat_back_addr":"172.21.15.124:6831/90894","heartbeat_front_addr":"172.21.15.124:6829/90894","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.827704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.839951+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.032230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.050605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.987908+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.031290+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.080707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.053692+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T01:41:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.480+0000 7f385cf33700 1 -- 172.21.15.27:0/3431811138 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3838066f30 msgr2=0x7f38380693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.480+0000 7f385cf33700 1 --2- 172.21.15.27:0/3431811138 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3838066f30 0x7f38380693e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f38400058d0 tx=0x7f3840009040 comp rx=0 tx=0).stop 2023-10-06T01:41:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.480+0000 7f385cf33700 1 -- 172.21.15.27:0/3431811138 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38581591b0 msgr2=0x7f385815b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.480+0000 7f385cf33700 1 --2- 172.21.15.27:0/3431811138 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38581591b0 0x7f385815b0f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f385000ec80 tx=0x7f38500091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.480+0000 7f385cf33700 1 -- 172.21.15.27:0/3431811138 shutdown_connections 2023-10-06T01:41:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.480+0000 7f385cf33700 1 --2- 172.21.15.27:0/3431811138 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3838066f30 0x7f38380693e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.480+0000 7f385cf33700 1 --2- 172.21.15.27:0/3431811138 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f385815d930 0x7f3858154780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.480+0000 7f385cf33700 1 --2- 172.21.15.27:0/3431811138 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f38581591b0 0x7f385815b0f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.480+0000 7f385cf33700 1 --2- 172.21.15.27:0/3431811138 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f385815d560 0x7f3858145020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.480+0000 7f385cf33700 1 -- 172.21.15.27:0/3431811138 >> 172.21.15.27:0/3431811138 conn(0x7f38580d1180 msgr2=0x7f38580b15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:23.481 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.480+0000 7f385cf33700 1 -- 172.21.15.27:0/3431811138 shutdown_connections 2023-10-06T01:41:23.482 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.480+0000 7f385cf33700 1 -- 172.21.15.27:0/3431811138 wait complete. 2023-10-06T01:41:23.493 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-06T01:41:23.493 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-06T01:41:23.494 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-06T01:41:23.494 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-10-06T01:41:23.494 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-10-06T01:41:23.495 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-10-06T01:41:23.495 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-10-06T01:41:23.495 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-10-06T01:41:23.700 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.699+0000 7f4f35cb6700 1 Processor -- start 2023-10-06T01:41:23.700 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.699+0000 7f4f35cb6700 1 -- start start 2023-10-06T01:41:23.701 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.701+0000 7f4f35cb6700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f4f300c9380 0x7f4f3006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.701 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.701+0000 7f4f35cb6700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4f300c7b70 0x7f4f3006c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.701 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.701+0000 7f4f35cb6700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f3015d5b0 con 0x7f4f300c9750 2023-10-06T01:41:23.701 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.701+0000 7f4f35cb6700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f3015d3a0 con 0x7f4f300c7b70 2023-10-06T01:41:23.702 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.701+0000 7f4f35cb6700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f3015d890 con 0x7f4f300c9380 2023-10-06T01:41:23.702 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.702+0000 7fc72a3a6700 1 Processor -- start 2023-10-06T01:41:23.703 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.702+0000 7fc72a3a6700 1 -- start start 2023-10-06T01:41:23.703 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.702+0000 7fd921dd6700 1 Processor -- start 2023-10-06T01:41:23.704 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.701+0000 7f4f2effd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4f300c7b70 0x7f4f3006c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.704 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.702+0000 7f4f2effd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4f300c7b70 0x7f4f3006c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55868/0 (socket says 172.21.15.27:55868) 2023-10-06T01:41:23.704 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.702+0000 7f4f2effd700 1 -- 172.21.15.27:0/667253622 learned_addr learned my addr 172.21.15.27:0/667253622 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:23.704 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.702+0000 7f4f2effd700 1 -- 172.21.15.27:0/667253622 >> v2:172.21.15.124:3300/0 conn(0x7f4f300c9380 msgr2=0x7f4f3006c420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-06T01:41:23.704 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.702+0000 7f4f2effd700 1 --2- 172.21.15.27:0/667253622 >> v2:172.21.15.124:3300/0 conn(0x7f4f300c9380 0x7f4f3006c420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.704 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.702+0000 7f4f2effd700 1 -- 172.21.15.27:0/667253622 >> v1:172.21.15.124:6789/0 conn(0x7f4f300c9750 legacy=0x7f4f301590e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.705 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.702+0000 7f4f2effd700 1 -- 172.21.15.27:0/667253622 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f3015db70 con 0x7f4f300c7b70 2023-10-06T01:41:23.705 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.702+0000 7f4f2effd700 1 --2- 172.21.15.27:0/667253622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f300c7b70 0x7f4f3006c960 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4f300cd280 tx=0x7f4f20009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=181c7b0b0b328c90 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:23.705 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.703+0000 7fd921dd6700 1 -- start start 2023-10-06T01:41:23.706 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.703+0000 7fc72a3a6700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc72415df30 0x7fc7240af750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.706 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.703+0000 7fc72a3a6700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7240b1d90 con 0x7fc72415db60 2023-10-06T01:41:23.707 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.703+0000 7fc72a3a6700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7240b1460 con 0x7fc72406b2e0 2023-10-06T01:41:23.707 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.703+0000 7fc72a3a6700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7240b1740 con 0x7fc72415df30 2023-10-06T01:41:23.707 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.703+0000 7fc728943700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc72415df30 0x7fc7240af750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.707 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.704+0000 7fc728943700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc72415df30 0x7fc7240af750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55876/0 (socket says 172.21.15.27:55876) 2023-10-06T01:41:23.707 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.704+0000 7fc728943700 1 -- 172.21.15.27:0/1281025174 learned_addr learned my addr 172.21.15.27:0/1281025174 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:23.707 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.703+0000 7f4f2e7fc700 1 -- 172.21.15.27:0/667253622 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f2000b860 con 0x7f4f300c7b70 2023-10-06T01:41:23.707 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.703+0000 7f4f2e7fc700 1 -- 172.21.15.27:0/667253622 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f2000b9c0 con 0x7f4f300c7b70 2023-10-06T01:41:23.707 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.703+0000 7f4f2e7fc700 1 -- 172.21.15.27:0/667253622 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f2000bd60 con 0x7f4f300c7b70 2023-10-06T01:41:23.708 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.704+0000 7fc728943700 1 -- 172.21.15.27:0/1281025174 >> v1:172.21.15.124:6789/0 conn(0x7fc72406b2e0 legacy=0x7fc7240af040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.708 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.704+0000 7fc728943700 1 -- 172.21.15.27:0/1281025174 >> v1:172.21.15.27:6789/0 conn(0x7fc72415db60 legacy=0x7fc72406abd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-06T01:41:23.708 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.704+0000 7fc728943700 1 -- 172.21.15.27:0/1281025174 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7240b1a20 con 0x7fc72415df30 2023-10-06T01:41:23.708 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.704+0000 7fd921dd6700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd91c0ab3b0 0x7fd91c156220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.708 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.704+0000 7fd921dd6700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd91c0adf70 con 0x7fd91c0cb980 2023-10-06T01:41:23.708 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.704+0000 7fd921dd6700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd91c0ad360 con 0x7fd91c0cbd50 2023-10-06T01:41:23.708 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.704+0000 7fd921dd6700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd91c0b0e00 con 0x7fd91c0ab3b0 2023-10-06T01:41:23.709 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.703+0000 7fd56cba1700 1 Processor -- start 2023-10-06T01:41:23.709 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.703+0000 7fd56cba1700 1 -- start start 2023-10-06T01:41:23.709 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.704+0000 7fd91affd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd91c0ab3b0 0x7fd91c156220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.709 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.704+0000 7fd91affd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd91c0ab3b0 0x7fd91c156220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55888/0 (socket says 172.21.15.27:55888) 2023-10-06T01:41:23.709 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.704+0000 7fd91affd700 1 -- 172.21.15.27:0/1228774094 learned_addr learned my addr 172.21.15.27:0/1228774094 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:23.709 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.704+0000 7fc728943700 1 --2- 172.21.15.27:0/1281025174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc72415df30 0x7fc7240af750 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc71c009f20 tx=0x7fc71c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d60a744438881558 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:23.710 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.705+0000 7fd91affd700 1 -- 172.21.15.27:0/1228774094 >> v1:172.21.15.124:6789/0 conn(0x7fd91c0cbd50 legacy=0x7fd91c0b15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:23.710 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.705+0000 7fd91affd700 1 -- 172.21.15.27:0/1228774094 >> v1:172.21.15.27:6789/0 conn(0x7fd91c0cb980 legacy=0x7fd91c0ae400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:23.710 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.705+0000 7fd91affd700 1 -- 172.21.15.27:0/1228774094 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd91c0ad640 con 0x7fd91c0ab3b0 2023-10-06T01:41:23.710 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.705+0000 7fd56cba1700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd56806b2e0 0x7fd5680af040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.710 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.705+0000 7fd56cba1700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fd56815df30 0x7fd5680af580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.710 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.705+0000 7fd56cba1700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5680b1d90 con 0x7fd56815db60 2023-10-06T01:41:23.711 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.705+0000 7fd56cba1700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5680afac0 con 0x7fd56815df30 2023-10-06T01:41:23.711 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.705+0000 7fd56cba1700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5680b1460 con 0x7fd56806b2e0 2023-10-06T01:41:23.711 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.705+0000 7fd91affd700 1 --2- 172.21.15.27:0/1228774094 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd91c0ab3b0 0x7fd91c156220 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd9100089b0 tx=0x7fd9100082d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eb3789e369a5a76 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:23.711 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.703+0000 7f52ae1b1700 1 Processor -- start 2023-10-06T01:41:23.712 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.703+0000 7f52ae1b1700 1 -- start start 2023-10-06T01:41:23.712 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.704+0000 7f52ae1b1700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f52a806ae20 0x7f52a80b1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.712 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.704+0000 7f52ae1b1700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52a80b1650 con 0x7f52a80acd70 2023-10-06T01:41:23.712 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.704+0000 7f52ae1b1700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52a80abbe0 con 0x7f52a806b1f0 2023-10-06T01:41:23.712 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.704+0000 7f52ae1b1700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52a80b1880 con 0x7f52a806ae20 2023-10-06T01:41:23.712 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.705+0000 7f52a6ffd700 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f52a806b1f0 0x7f52a80ae6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.27:46160/0 (socket says 172.21.15.27:46160) 2023-10-06T01:41:23.712 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.705+0000 7f52a6ffd700 1 -- 172.21.15.27:0/2409681310 learned_addr learned my addr 172.21.15.27:0/2409681310 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:23.713 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.705+0000 7fc722ffd700 1 -- 172.21.15.27:0/1281025174 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71c00e070 con 0x7fc72415df30 2023-10-06T01:41:23.713 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.705+0000 7fc722ffd700 1 -- 172.21.15.27:0/1281025174 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc71c00ba40 con 0x7fc72415df30 2023-10-06T01:41:23.713 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.705+0000 7fc722ffd700 1 -- 172.21.15.27:0/1281025174 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71c0092e0 con 0x7fc72415df30 2023-10-06T01:41:23.713 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.705+0000 7fd56659c700 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fd56815db60 0x7fd56806abd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.27:46174/0 (socket says 172.21.15.27:46174) 2023-10-06T01:41:23.713 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.705+0000 7fd56659c700 1 -- 172.21.15.27:0/2789087444 learned_addr learned my addr 172.21.15.27:0/2789087444 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:23.713 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.705+0000 7fd566d9d700 1 --2- 172.21.15.27:0/2789087444 >> v2:172.21.15.124:3300/0 conn(0x7fd56815df30 0x7fd5680af580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.713 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.705+0000 7f52a77fe700 1 --2- 172.21.15.27:0/2409681310 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f52a806ae20 0x7f52a80b1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.714 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.705+0000 7fd91a7fc700 1 -- 172.21.15.27:0/1228774094 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd910010070 con 0x7fd91c0ab3b0 2023-10-06T01:41:23.714 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.705+0000 7fd91a7fc700 1 -- 172.21.15.27:0/1228774094 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd91000daf0 con 0x7fd91c0ab3b0 2023-10-06T01:41:23.714 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.705+0000 7fd565d9b700 1 --2- 172.21.15.27:0/2789087444 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd56806b2e0 0x7fd5680af040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.714 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.705+0000 7f4f35cb6700 1 -- 172.21.15.27:0/667253622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f300c7b70 msgr2=0x7f4f3006c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.715 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.705+0000 7f4f35cb6700 1 --2- 172.21.15.27:0/667253622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f300c7b70 0x7f4f3006c960 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4f300cd280 tx=0x7f4f20009250 comp rx=0 tx=0).stop 2023-10-06T01:41:23.715 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.706+0000 7f52a67fc700 1 -- 172.21.15.27:0/2409681310 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3442148002 0 0) 0x7f52a80abbe0 con 0x7f52a806b1f0 2023-10-06T01:41:23.715 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.706+0000 7f52a67fc700 1 -- 172.21.15.27:0/2409681310 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5290003570 con 0x7f52a806b1f0 2023-10-06T01:41:23.715 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.706+0000 7fd565d9b700 1 -- 172.21.15.27:0/2789087444 >> v2:172.21.15.124:3300/0 conn(0x7fd56815df30 msgr2=0x7fd5680af580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:23.715 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.706+0000 7fd565d9b700 1 --2- 172.21.15.27:0/2789087444 >> v2:172.21.15.124:3300/0 conn(0x7fd56815df30 0x7fd5680af580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.715 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.706+0000 7fd565d9b700 1 -- 172.21.15.27:0/2789087444 >> v1:172.21.15.124:6789/0 conn(0x7fd56815db60 legacy=0x7fd56806abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.716 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.706+0000 7fd565d9b700 1 -- 172.21.15.27:0/2789087444 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5680b1740 con 0x7fd56806b2e0 2023-10-06T01:41:23.716 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.706+0000 7f52a67fc700 1 -- 172.21.15.27:0/2409681310 <== mon.1 v1:172.21.15.124:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3729144082 0 0) 0x7f5290003570 con 0x7f52a806b1f0 2023-10-06T01:41:23.716 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.706+0000 7f52a67fc700 1 -- 172.21.15.27:0/2409681310 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f52a806ae20 msgr2=0x7f52a80b1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:23.716 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.706+0000 7f52a67fc700 1 --2- 172.21.15.27:0/2409681310 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f52a806ae20 0x7f52a80b1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.716 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.706+0000 7f52a67fc700 1 -- 172.21.15.27:0/2409681310 >> v1:172.21.15.27:6789/0 conn(0x7f52a80acd70 legacy=0x7f52a80ac0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:23.716 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.706+0000 7f52a67fc700 1 -- 172.21.15.27:0/2409681310 --> v1:172.21.15.124:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a80b1b60 con 0x7f52a806b1f0 2023-10-06T01:41:23.717 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.706+0000 7f52a67fc700 1 -- 172.21.15.27:0/2409681310 <== mon.1 v1:172.21.15.124:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2619078944 0 0) 0x7f529c001470 con 0x7f52a806b1f0 2023-10-06T01:41:23.717 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.706+0000 7f52a67fc700 1 -- 172.21.15.27:0/2409681310 >> v1:172.21.15.124:6789/0 conn(0x7f52a806b1f0 legacy=0x7f52a80ae6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.717 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.706+0000 7f52a67fc700 1 --2- 172.21.15.27:0/2409681310 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5290003a90 0x7f5290003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.717 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.706+0000 7f52a67fc700 1 --2- 172.21.15.27:0/2409681310 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5290004810 0x7f5290004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.717 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.706+0000 7f52a67fc700 1 --2- 172.21.15.27:0/2409681310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52900051c0 0x7f529000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.717 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.706+0000 7f52a67fc700 1 -- 172.21.15.27:0/2409681310 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a80b1880 con 0x7f52900051c0 2023-10-06T01:41:23.717 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.706+0000 7f52a67fc700 1 -- 172.21.15.27:0/2409681310 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f529000bfd0 con 0x7f5290003a90 2023-10-06T01:41:23.717 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.706+0000 7f52a67fc700 1 -- 172.21.15.27:0/2409681310 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f529000c190 con 0x7f5290004810 2023-10-06T01:41:23.717 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.706+0000 7fc72a3a6700 1 -- 172.21.15.27:0/1281025174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc72415df30 msgr2=0x7fc7240af750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.718 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.706+0000 7fc72a3a6700 1 --2- 172.21.15.27:0/1281025174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc72415df30 0x7fc7240af750 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc71c009f20 tx=0x7fc71c009840 comp rx=0 tx=0).stop 2023-10-06T01:41:23.718 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.706+0000 7f52a6ffd700 1 --2- 172.21.15.27:0/2409681310 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5290004810 0x7f5290004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.718 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.706+0000 7fd565d9b700 1 --2- 172.21.15.27:0/2789087444 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd56806b2e0 0x7fd5680af040 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd55c0098e0 tx=0x7fd55c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d63de6859ae85fa server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:23.719 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.706+0000 7fc72a3a6700 1 -- 172.21.15.27:0/1281025174 shutdown_connections 2023-10-06T01:41:23.719 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.706+0000 7fc72a3a6700 1 --2- 172.21.15.27:0/1281025174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc72415df30 0x7fc7240af750 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.719 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.706+0000 7fc72a3a6700 1 -- 172.21.15.27:0/1281025174 >> 172.21.15.27:0/1281025174 conn(0x7fc724068b90 msgr2=0x7fc7240694b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:23.719 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.705+0000 7fd91a7fc700 1 -- 172.21.15.27:0/1228774094 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91000de90 con 0x7fd91c0ab3b0 2023-10-06T01:41:23.719 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.706+0000 7fd921dd6700 1 -- 172.21.15.27:0/1228774094 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd91c0ab3b0 msgr2=0x7fd91c156220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.719 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.706+0000 7fd921dd6700 1 --2- 172.21.15.27:0/1228774094 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd91c0ab3b0 0x7fd91c156220 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd9100089b0 tx=0x7fd9100082d0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.720 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.706+0000 7fd56559a700 1 -- 172.21.15.27:0/2789087444 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd55c009e80 con 0x7fd56806b2e0 2023-10-06T01:41:23.720 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.706+0000 7fd56559a700 1 -- 172.21.15.27:0/2789087444 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd55c00b860 con 0x7fd56806b2e0 2023-10-06T01:41:23.720 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.706+0000 7fd56559a700 1 -- 172.21.15.27:0/2789087444 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd55c00bc20 con 0x7fd56806b2e0 2023-10-06T01:41:23.720 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.706+0000 7fc72a3a6700 1 -- 172.21.15.27:0/1281025174 shutdown_connections 2023-10-06T01:41:23.720 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.706+0000 7fc72a3a6700 1 -- 172.21.15.27:0/1281025174 wait complete. 2023-10-06T01:41:23.721 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.706+0000 7f52a6ffd700 1 -- 172.21.15.27:0/2409681310 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5290003a90 msgr2=0x7f5290003e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-06T01:41:23.721 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.707+0000 7f52a6ffd700 1 --2- 172.21.15.27:0/2409681310 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5290003a90 0x7f5290003e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.721 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.707+0000 7f52a6ffd700 1 -- 172.21.15.27:0/2409681310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52900051c0 msgr2=0x7f529000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:23.721 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.707+0000 7f52a6ffd700 1 --2- 172.21.15.27:0/2409681310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f52900051c0 0x7f529000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.721 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.707+0000 7f52a6ffd700 1 -- 172.21.15.27:0/2409681310 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a80b1b60 con 0x7f5290004810 2023-10-06T01:41:23.721 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.707+0000 7f52a6ffd700 1 --2- 172.21.15.27:0/2409681310 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5290004810 0x7f5290004c80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f529c00c460 tx=0x7f529c0079d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=762ec2df173e32ee server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:23.721 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.707+0000 7f52a67fc700 1 -- 172.21.15.27:0/2409681310 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529c00cd90 con 0x7f5290004810 2023-10-06T01:41:23.721 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.707+0000 7f52a67fc700 1 -- 172.21.15.27:0/2409681310 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f529c005920 con 0x7f5290004810 2023-10-06T01:41:23.721 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.707+0000 7f52a67fc700 1 -- 172.21.15.27:0/2409681310 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529c005ce0 con 0x7f5290004810 2023-10-06T01:41:23.722 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.707+0000 7f4f35cb6700 1 -- 172.21.15.27:0/667253622 shutdown_connections 2023-10-06T01:41:23.722 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.707+0000 7f4f35cb6700 1 --2- 172.21.15.27:0/667253622 >> v2:172.21.15.124:3300/0 conn(0x7f4f300c9380 0x7f4f3006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.722 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.707+0000 7f4f35cb6700 1 --2- 172.21.15.27:0/667253622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f300c7b70 0x7f4f3006c960 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.722 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.707+0000 7f4f35cb6700 1 -- 172.21.15.27:0/667253622 >> 172.21.15.27:0/667253622 conn(0x7f4f3006e5f0 msgr2=0x7f4f3006e9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:23.722 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.707+0000 7fd56cba1700 1 -- 172.21.15.27:0/2789087444 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd56806b2e0 msgr2=0x7fd5680af040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.722 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.707+0000 7fd56cba1700 1 --2- 172.21.15.27:0/2789087444 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd56806b2e0 0x7fd5680af040 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd55c0098e0 tx=0x7fd55c0091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.723 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.707+0000 7fd921dd6700 1 -- 172.21.15.27:0/1228774094 shutdown_connections 2023-10-06T01:41:23.723 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.707+0000 7fd921dd6700 1 --2- 172.21.15.27:0/1228774094 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd91c0ab3b0 0x7fd91c156220 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.723 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.707+0000 7fd921dd6700 1 -- 172.21.15.27:0/1228774094 >> 172.21.15.27:0/1228774094 conn(0x7fd91c068b90 msgr2=0x7fd91c0694b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:23.723 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.708+0000 7fd56cba1700 1 -- 172.21.15.27:0/2789087444 shutdown_connections 2023-10-06T01:41:23.724 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.708+0000 7fd56cba1700 1 --2- 172.21.15.27:0/2789087444 >> v2:172.21.15.124:3300/0 conn(0x7fd56815df30 0x7fd5680af580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.724 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.708+0000 7fd56cba1700 1 --2- 172.21.15.27:0/2789087444 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd56806b2e0 0x7fd5680af040 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.724 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.708+0000 7fd56cba1700 1 -- 172.21.15.27:0/2789087444 >> 172.21.15.27:0/2789087444 conn(0x7fd568068b90 msgr2=0x7fd5680694b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:23.724 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.708+0000 7f4f35cb6700 1 -- 172.21.15.27:0/667253622 shutdown_connections 2023-10-06T01:41:23.725 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.708+0000 7fd921dd6700 1 -- 172.21.15.27:0/1228774094 shutdown_connections 2023-10-06T01:41:23.725 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.708+0000 7fd921dd6700 1 -- 172.21.15.27:0/1228774094 wait complete. 2023-10-06T01:41:23.725 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.708+0000 7f4f35cb6700 1 -- 172.21.15.27:0/667253622 wait complete. 2023-10-06T01:41:23.725 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.710+0000 7f4f35cb6700 1 Processor -- start 2023-10-06T01:41:23.725 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.708+0000 7fc72a3a6700 1 Processor -- start 2023-10-06T01:41:23.726 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.710+0000 7fd921dd6700 1 Processor -- start 2023-10-06T01:41:23.726 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.709+0000 7f18ee686700 1 Processor -- start 2023-10-06T01:41:23.726 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.708+0000 7f52ae1b1700 1 -- 172.21.15.27:0/2409681310 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5290004810 msgr2=0x7f5290004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.726 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.708+0000 7f52ae1b1700 1 --2- 172.21.15.27:0/2409681310 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5290004810 0x7f5290004c80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f529c00c460 tx=0x7f529c0079d0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.726 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.708+0000 7f52a7fff700 1 -- 172.21.15.27:0/2409681310 reap_dead start 2023-10-06T01:41:23.727 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.708+0000 7f52ae1b1700 1 -- 172.21.15.27:0/2409681310 shutdown_connections 2023-10-06T01:41:23.727 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.708+0000 7f52ae1b1700 1 -- 172.21.15.27:0/2409681310 >> 172.21.15.27:0/2409681310 conn(0x7f52a8068b90 msgr2=0x7f52a80694b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:23.727 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.708+0000 7f52ae1b1700 1 -- 172.21.15.27:0/2409681310 shutdown_connections 2023-10-06T01:41:23.727 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.708+0000 7fd56cba1700 1 -- 172.21.15.27:0/2789087444 shutdown_connections 2023-10-06T01:41:23.727 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.708+0000 7fd56cba1700 1 -- 172.21.15.27:0/2789087444 wait complete. 2023-10-06T01:41:23.727 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.708+0000 7fc72a3a6700 1 -- start start 2023-10-06T01:41:23.728 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.710+0000 7fd56cba1700 1 Processor -- start 2023-10-06T01:41:23.728 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.710+0000 7fc72a3a6700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc72406b2e0 0x7fc7240a6280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.728 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.710+0000 7fc72a3a6700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc72415db60 0x7fc72415c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.728 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.710+0000 7fc72a3a6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc72415df30 0x7fc7240ac050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.728 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.710+0000 7fc72a3a6700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc72415f740 con 0x7fc72415df30 2023-10-06T01:41:23.728 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.710+0000 7fc72a3a6700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc724171a00 con 0x7fc72406b2e0 2023-10-06T01:41:23.728 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.710+0000 7fc72a3a6700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc724171ce0 con 0x7fc72415db60 2023-10-06T01:41:23.729 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.710+0000 7fc728943700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc72415df30 0x7fc7240ac050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.729 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.710+0000 7fc728943700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc72415df30 0x7fc7240ac050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55900/0 (socket says 172.21.15.27:55900) 2023-10-06T01:41:23.729 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.710+0000 7fc728943700 1 -- 172.21.15.27:0/3831465769 learned_addr learned my addr 172.21.15.27:0/3831465769 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:23.729 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.710+0000 7fd56cba1700 1 -- start start 2023-10-06T01:41:23.729 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.708+0000 7f52ae1b1700 1 -- 172.21.15.27:0/2409681310 wait complete. 2023-10-06T01:41:23.729 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.710+0000 7f52ae1b1700 1 Processor -- start 2023-10-06T01:41:23.730 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.710+0000 7fc723fff700 1 --2- 172.21.15.27:0/3831465769 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc72406b2e0 0x7fc7240a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.730 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.710+0000 7fc7237fe700 1 --2- 172.21.15.27:0/3831465769 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc72415db60 0x7fc72415c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.730 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7fc723fff700 1 -- 172.21.15.27:0/3831465769 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc72415db60 msgr2=0x7fc72415c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.730 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7fc723fff700 1 --2- 172.21.15.27:0/3831465769 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc72415db60 0x7fc72415c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.730 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7fc723fff700 1 -- 172.21.15.27:0/3831465769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc72415df30 msgr2=0x7fc7240ac050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7fc723fff700 1 --2- 172.21.15.27:0/3831465769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc72415df30 0x7fc7240ac050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7fc723fff700 1 -- 172.21.15.27:0/3831465769 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc71c009ae0 con 0x7fc72406b2e0 2023-10-06T01:41:23.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.709+0000 7f18ee686700 1 -- start start 2023-10-06T01:41:23.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7f18ee686700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f18e8068890 0x7f18e80ae010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7f18ee686700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f18e80afb90 0x7f18e815d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7f18ee686700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e80ae550 con 0x7f18e8068c60 2023-10-06T01:41:23.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7f18ee686700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18e80aea40 con 0x7f18e8068890 2023-10-06T01:41:23.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7f18ee686700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18e80adb80 con 0x7f18e80afb90 2023-10-06T01:41:23.732 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7fc723fff700 1 --2- 172.21.15.27:0/3831465769 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc72406b2e0 0x7fc7240a6280 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc71400b530 tx=0x7fc7140090f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:23.732 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7fd56cba1700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd56806b2e0 0x7fd56815f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.732 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7fd56cba1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd56815db60 0x7fd56815fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.732 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7fd56cba1700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd56815df30 0x7fd5680cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.732 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7fd56cba1700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5680ca100 con 0x7fd56815db60 2023-10-06T01:41:23.732 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7fd56cba1700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd568160520 con 0x7fd56815df30 2023-10-06T01:41:23.732 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7fd56cba1700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5680ca980 con 0x7fd56806b2e0 2023-10-06T01:41:23.733 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7f18ecc23700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f18e80afb90 0x7f18e815d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.733 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7f18ecc23700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f18e80afb90 0x7f18e815d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.27:43082/0 (socket says 172.21.15.27:43082) 2023-10-06T01:41:23.733 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7f18ecc23700 1 -- 172.21.15.27:0/898239276 learned_addr learned my addr 172.21.15.27:0/898239276 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:23.733 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7f18ecc23700 1 -- 172.21.15.27:0/898239276 >> v2:172.21.15.27:3300/0 conn(0x7f18e8068890 msgr2=0x7f18e80ae010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:23.733 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7f18ecc23700 1 --2- 172.21.15.27:0/898239276 >> v2:172.21.15.27:3300/0 conn(0x7f18e8068890 0x7f18e80ae010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.733 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7f18ecc23700 1 -- 172.21.15.27:0/898239276 >> v1:172.21.15.124:6789/0 conn(0x7f18e8068c60 legacy=0x7f18e80af480 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-06T01:41:23.733 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7f18ecc23700 1 -- 172.21.15.27:0/898239276 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e80b00d0 con 0x7f18e80afb90 2023-10-06T01:41:23.734 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7fd566d9d700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd56815df30 0x7fd5680cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.734 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7fd566d9d700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd56815df30 0x7fd5680cae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.27:43098/0 (socket says 172.21.15.27:43098) 2023-10-06T01:41:23.734 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7fd566d9d700 1 -- 172.21.15.27:0/3621683566 learned_addr learned my addr 172.21.15.27:0/3621683566 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:23.734 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.710+0000 7f52ae1b1700 1 -- start start 2023-10-06T01:41:23.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7f52ae1b1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5290004810 0x7f52a8096a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7f52ae1b1700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f52a80c7500 0x7f52a817f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7f52ae1b1700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f52a8098060 0x7f52a816bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7f52ae1b1700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a80c8900 con 0x7f5290004810 2023-10-06T01:41:23.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7f52ae1b1700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a8097040 con 0x7f52a80c7500 2023-10-06T01:41:23.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7f52ae1b1700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52a8097320 con 0x7f52a8098060 2023-10-06T01:41:23.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7f52a77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5290004810 0x7f52a8096a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7f52a77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5290004810 0x7f52a8096a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55934/0 (socket says 172.21.15.27:55934) 2023-10-06T01:41:23.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7f52a77fe700 1 -- 172.21.15.27:0/807096333 learned_addr learned my addr 172.21.15.27:0/807096333 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:23.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7fd565d9b700 1 --2- 172.21.15.27:0/3621683566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd56815db60 0x7fd56815fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.736 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7f52a6ffd700 1 --2- 172.21.15.27:0/807096333 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f52a80c7500 0x7f52a817f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.736 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7f52a7fff700 1 --2- 172.21.15.27:0/807096333 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f52a8098060 0x7f52a816bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.736 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.710+0000 7fd921dd6700 1 -- start start 2023-10-06T01:41:23.736 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7fd921dd6700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd91c0ab3b0 0x7fd91c14e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.736 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7fd921dd6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd91c0cb980 0x7fd91c0a8a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.737 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7fd566d9d700 1 -- 172.21.15.27:0/3621683566 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd56806b2e0 msgr2=0x7fd56815f570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-06T01:41:23.737 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7fd566d9d700 1 --2- 172.21.15.27:0/3621683566 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd56806b2e0 0x7fd56815f570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.737 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7fd566d9d700 1 -- 172.21.15.27:0/3621683566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd56815db60 msgr2=0x7fd56815fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.737 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7fd566d9d700 1 --2- 172.21.15.27:0/3621683566 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd56815db60 0x7fd56815fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.737 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7fd921dd6700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd91c0cbd50 0x7fd91c149970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.737 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7fd921dd6700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91c14ea30 con 0x7fd91c0cb980 2023-10-06T01:41:23.737 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7fd921dd6700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91c0a8fc0 con 0x7fd91c0cbd50 2023-10-06T01:41:23.737 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7fd921dd6700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd91c171a90 con 0x7fd91c0ab3b0 2023-10-06T01:41:23.738 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7fd91b7fe700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd91c0ab3b0 0x7fd91c14e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.738 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7fd91affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd91c0cb980 0x7fd91c0a8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.738 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7fd566d9d700 1 -- 172.21.15.27:0/3621683566 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd55c009420 con 0x7fd56815df30 2023-10-06T01:41:23.738 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7fd91affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd91c0cb980 0x7fd91c0a8a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55926/0 (socket says 172.21.15.27:55926) 2023-10-06T01:41:23.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7fd91affd700 1 -- 172.21.15.27:0/1482986497 learned_addr learned my addr 172.21.15.27:0/1482986497 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:23.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7fd91bfff700 1 --2- 172.21.15.27:0/1482986497 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd91c0cbd50 0x7fd91c149970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7fd91affd700 1 -- 172.21.15.27:0/1482986497 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd91c0ab3b0 msgr2=0x7fd91c14e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7fd91affd700 1 --2- 172.21.15.27:0/1482986497 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd91c0ab3b0 0x7fd91c14e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7fd91affd700 1 -- 172.21.15.27:0/1482986497 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd91c0cbd50 msgr2=0x7fd91c149970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7fd91affd700 1 --2- 172.21.15.27:0/1482986497 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd91c0cbd50 0x7fd91c149970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7fd91affd700 1 -- 172.21.15.27:0/1482986497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd910008570 con 0x7fd91c0cb980 2023-10-06T01:41:23.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7fd91bfff700 1 --2- 172.21.15.27:0/1482986497 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd91c0cbd50 0x7fd91c149970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:41:23.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7f18ecc23700 1 --2- 172.21.15.27:0/898239276 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18e80afb90 0x7f18e815d690 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f18e806d790 tx=0x7f18e0009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6e6e3f14565e53c server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:23.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7f52a6ffd700 1 -- 172.21.15.27:0/807096333 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f52a8098060 msgr2=0x7f52a816bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7f52a6ffd700 1 --2- 172.21.15.27:0/807096333 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f52a8098060 0x7f52a816bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7f52a6ffd700 1 -- 172.21.15.27:0/807096333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5290004810 msgr2=0x7f52a8096a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7f52a6ffd700 1 --2- 172.21.15.27:0/807096333 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5290004810 0x7f52a8096a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7f52a6ffd700 1 -- 172.21.15.27:0/807096333 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a8097600 con 0x7f52a80c7500 2023-10-06T01:41:23.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7fd566d9d700 1 --2- 172.21.15.27:0/3621683566 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd56815df30 0x7fd5680cae10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd55800d890 tx=0x7fd558008040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:23.741 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.711+0000 7f4f35cb6700 1 -- start start 2023-10-06T01:41:23.741 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7f4f35cb6700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4f300c7b70 0x7f4f301520f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.741 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7f4f35cb6700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4f300c9380 0x7f4f300b0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.741 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7f4f35cb6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f300c9750 0x7f4f30175e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.741 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7f4f35cb6700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f300b15b0 con 0x7f4f300c9750 2023-10-06T01:41:23.741 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7f18e6ffd700 1 -- 172.21.15.27:0/898239276 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e000e070 con 0x7f18e80afb90 2023-10-06T01:41:23.742 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7f4f35cb6700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f300b1890 con 0x7f4f300c7b70 2023-10-06T01:41:23.742 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7f4f35cb6700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f300b1b70 con 0x7f4f300c9380 2023-10-06T01:41:23.742 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7f18e6ffd700 1 -- 172.21.15.27:0/898239276 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18e000ba10 con 0x7f18e80afb90 2023-10-06T01:41:23.742 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7f18e6ffd700 1 -- 172.21.15.27:0/898239276 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18e00092e0 con 0x7f18e80afb90 2023-10-06T01:41:23.742 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7fd91affd700 1 --2- 172.21.15.27:0/1482986497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd91c0cb980 0x7fd91c0a8a80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd910008880 tx=0x7fd91001b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:23.743 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.713+0000 7f4f2effd700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4f300c9380 0x7f4f300b0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.743 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.713+0000 7f4f2effd700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4f300c9380 0x7f4f300b0b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.27:35960/0 (socket says 172.21.15.27:35960) 2023-10-06T01:41:23.743 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.713+0000 7f4f2effd700 1 -- 172.21.15.27:0/556383606 learned_addr learned my addr 172.21.15.27:0/556383606 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:23.743 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.713+0000 7f52a6ffd700 1 --2- 172.21.15.27:0/807096333 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f52a80c7500 0x7f52a817f470 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f529c014000 tx=0x7f529c015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:23.744 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.713+0000 7f4f2ffff700 1 --2- 172.21.15.27:0/556383606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f300c9750 0x7f4f30175e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.744 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.713+0000 7f4f2f7fe700 1 --2- 172.21.15.27:0/556383606 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4f300c7b70 0x7f4f301520f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.744 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.713+0000 7f4f2effd700 1 -- 172.21.15.27:0/556383606 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4f300c7b70 msgr2=0x7f4f301520f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.744 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.713+0000 7f4f2effd700 1 --2- 172.21.15.27:0/556383606 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4f300c7b70 0x7f4f301520f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.744 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.713+0000 7f4f2effd700 1 -- 172.21.15.27:0/556383606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f300c9750 msgr2=0x7f4f30175e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.744 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.713+0000 7f4f2effd700 1 --2- 172.21.15.27:0/556383606 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4f300c9750 0x7f4f30175e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.744 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.713+0000 7f4f2effd700 1 -- 172.21.15.27:0/556383606 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f200094f0 con 0x7f4f300c9380 2023-10-06T01:41:23.745 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.713+0000 7fd9037fe700 1 -- 172.21.15.27:0/1482986497 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd910016ba0 con 0x7fd91c0cb980 2023-10-06T01:41:23.745 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.713+0000 7fd9037fe700 1 -- 172.21.15.27:0/1482986497 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd910016d00 con 0x7fd91c0cb980 2023-10-06T01:41:23.745 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.713+0000 7fd9037fe700 1 -- 172.21.15.27:0/1482986497 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91000b310 con 0x7fd91c0cb980 2023-10-06T01:41:23.745 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.713+0000 7f4f2effd700 1 --2- 172.21.15.27:0/556383606 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4f300c9380 0x7f4f300b0b10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4f30063350 tx=0x7f4f20017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:23.745 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.713+0000 7fd921dd6700 1 -- 172.21.15.27:0/1482986497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd91c171d70 con 0x7fd91c0cb980 2023-10-06T01:41:23.746 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.713+0000 7fd921dd6700 1 -- 172.21.15.27:0/1482986497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd91c172330 con 0x7fd91c0cb980 2023-10-06T01:41:23.746 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.712+0000 7fd5567fc700 1 -- 172.21.15.27:0/3621683566 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5580125f0 con 0x7fd56815df30 2023-10-06T01:41:23.746 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.713+0000 7fd5567fc700 1 -- 172.21.15.27:0/3621683566 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd558012750 con 0x7fd56815df30 2023-10-06T01:41:23.746 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.713+0000 7fd5567fc700 1 -- 172.21.15.27:0/3621683566 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd558012b10 con 0x7fd56815df30 2023-10-06T01:41:23.746 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.713+0000 7fd56cba1700 1 -- 172.21.15.27:0/3621683566 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd56815fff0 con 0x7fd56815df30 2023-10-06T01:41:23.746 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.713+0000 7fd56cba1700 1 -- 172.21.15.27:0/3621683566 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd568160240 con 0x7fd56815df30 2023-10-06T01:41:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.714+0000 7f4f177fe700 1 -- 172.21.15.27:0/556383606 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f20011e40 con 0x7f4f300c9380 2023-10-06T01:41:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.714+0000 7f4f177fe700 1 -- 172.21.15.27:0/556383606 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f2001a420 con 0x7f4f300c9380 2023-10-06T01:41:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.714+0000 7f4f177fe700 1 -- 172.21.15.27:0/556383606 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f2001a7e0 con 0x7f4f300c9380 2023-10-06T01:41:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.714+0000 7fd56cba1700 1 -- 172.21.15.27:0/3621683566 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd568064050 con 0x7fd56815df30 2023-10-06T01:41:23.747 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.713+0000 7f528f7fe700 1 -- 172.21.15.27:0/807096333 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529c00ce60 con 0x7f52a80c7500 2023-10-06T01:41:23.748 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.713+0000 7f528f7fe700 1 -- 172.21.15.27:0/807096333 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f529c0074f0 con 0x7f52a80c7500 2023-10-06T01:41:23.748 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.713+0000 7f528f7fe700 1 -- 172.21.15.27:0/807096333 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529c01f410 con 0x7f52a80c7500 2023-10-06T01:41:23.748 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.713+0000 7f52ae1b1700 1 -- 172.21.15.27:0/807096333 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52a80978e0 con 0x7f52a80c7500 2023-10-06T01:41:23.748 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.713+0000 7f52ae1b1700 1 -- 172.21.15.27:0/807096333 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52a80c7970 con 0x7f52a80c7500 2023-10-06T01:41:23.748 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.713+0000 7f18ee686700 1 -- 172.21.15.27:0/898239276 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18e80afb90 msgr2=0x7f18e815d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.748 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.713+0000 7f18ee686700 1 --2- 172.21.15.27:0/898239276 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18e80afb90 0x7f18e815d690 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f18e806d790 tx=0x7f18e0009840 comp rx=0 tx=0).stop 2023-10-06T01:41:23.749 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.714+0000 7fc70bfff700 1 -- 172.21.15.27:0/3831465769 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7140059d0 con 0x7fc72406b2e0 2023-10-06T01:41:23.749 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.714+0000 7fc70bfff700 1 -- 172.21.15.27:0/3831465769 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc714005b50 con 0x7fc72406b2e0 2023-10-06T01:41:23.749 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.714+0000 7fc72a3a6700 1 -- 172.21.15.27:0/3831465769 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc724171fc0 con 0x7fc72406b2e0 2023-10-06T01:41:23.749 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.714+0000 7fc72a3a6700 1 -- 172.21.15.27:0/3831465769 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7240a34b0 con 0x7fc72406b2e0 2023-10-06T01:41:23.749 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.714+0000 7f52ae1b1700 1 -- 172.21.15.27:0/807096333 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f52a8064050 con 0x7f52a80c7500 2023-10-06T01:41:23.750 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.714+0000 7f18ee686700 1 -- 172.21.15.27:0/898239276 shutdown_connections 2023-10-06T01:41:23.750 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.714+0000 7f18ee686700 1 --2- 172.21.15.27:0/898239276 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18e80afb90 0x7f18e815d690 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.750 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.714+0000 7f18ee686700 1 --2- 172.21.15.27:0/898239276 >> v2:172.21.15.27:3300/0 conn(0x7f18e8068890 0x7f18e80ae010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.750 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.714+0000 7f18ee686700 1 -- 172.21.15.27:0/898239276 >> 172.21.15.27:0/898239276 conn(0x7f18e806be70 msgr2=0x7f18e80cc6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:23.750 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.714+0000 7f18ee686700 1 -- 172.21.15.27:0/898239276 shutdown_connections 2023-10-06T01:41:23.750 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.714+0000 7f18ee686700 1 -- 172.21.15.27:0/898239276 wait complete. 2023-10-06T01:41:23.750 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.714+0000 7f4f35cb6700 1 -- 172.21.15.27:0/556383606 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f300b1e50 con 0x7f4f300c9380 2023-10-06T01:41:23.750 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.714+0000 7f4f35cb6700 1 -- 172.21.15.27:0/556383606 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f3014dbb0 con 0x7f4f300c9380 2023-10-06T01:41:23.751 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.714+0000 7fd921dd6700 1 -- 172.21.15.27:0/1482986497 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd91c064050 con 0x7fd91c0cb980 2023-10-06T01:41:23.751 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.715+0000 7f528f7fe700 1 -- 172.21.15.27:0/807096333 <== mon.1 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f529c01f780 con 0x7f52a80c7500 2023-10-06T01:41:23.751 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.715+0000 7fc70bfff700 1 -- 172.21.15.27:0/3831465769 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc714006030 con 0x7fc72406b2e0 2023-10-06T01:41:23.751 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.715+0000 7f4f35cb6700 1 -- 172.21.15.27:0/556383606 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4f300b1a00 con 0x7f4f300c9380 2023-10-06T01:41:23.752 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.715+0000 7fbdecdb8700 1 Processor -- start 2023-10-06T01:41:23.752 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.715+0000 7fbdecdb8700 1 -- start start 2023-10-06T01:41:23.752 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.715+0000 7fc72a3a6700 1 -- 172.21.15.27:0/3831465769 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc724064050 con 0x7fc72406b2e0 2023-10-06T01:41:23.752 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.716+0000 7f18ee686700 1 Processor -- start 2023-10-06T01:41:23.753 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.716+0000 7f18ee686700 1 -- start start 2023-10-06T01:41:23.753 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.716+0000 7f528f7fe700 1 --2- 172.21.15.27:0/807096333 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5294066f30 0x7f52940693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.753 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.716+0000 7f528f7fe700 1 -- 172.21.15.27:0/807096333 <== mon.1 v2:172.21.15.124:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f529c086050 con 0x7f52a80c7500 2023-10-06T01:41:23.753 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.716+0000 7f528f7fe700 1 --2- 172.21.15.27:0/807096333 >> [v2:172.21.15.124:6800/90863,v1:172.21.15.124:6801/90863] conn(0x7f529406f760 0x7f5294071b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.753 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.716+0000 7f528f7fe700 1 -- 172.21.15.27:0/807096333 --> [v2:172.21.15.124:6800/90863,v1:172.21.15.124:6801/90863] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5294072220 con 0x7f529406f760 2023-10-06T01:41:23.753 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.716+0000 7f52a7fff700 1 --2- 172.21.15.27:0/807096333 >> [v2:172.21.15.124:6800/90863,v1:172.21.15.124:6801/90863] conn(0x7f529406f760 0x7f5294071b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.753 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.716+0000 7f52a77fe700 1 --2- 172.21.15.27:0/807096333 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5294066f30 0x7f52940693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.754 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.716+0000 7fd5567fc700 1 -- 172.21.15.27:0/3621683566 <== mon.1 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd55801e460 con 0x7fd56815df30 2023-10-06T01:41:23.754 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.717+0000 7f528f7fe700 1 -- 172.21.15.27:0/807096333 <== mon.1 v2:172.21.15.124:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f529c053160 con 0x7f52a80c7500 2023-10-06T01:41:23.754 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.717+0000 7fd5567fc700 1 --2- 172.21.15.27:0/3621683566 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd548067000 0x7fd5480694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.755 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.716+0000 7fd9037fe700 1 -- 172.21.15.27:0/1482986497 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd91001d040 con 0x7fd91c0cb980 2023-10-06T01:41:23.755 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.717+0000 7f52a77fe700 1 --2- 172.21.15.27:0/807096333 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5294066f30 0x7f52940693e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5298009fa0 tx=0x7f529800b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:23.755 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.717+0000 7fd5567fc700 1 -- 172.21.15.27:0/3621683566 <== mon.1 v2:172.21.15.124:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fd558085810 con 0x7fd56815df30 2023-10-06T01:41:23.755 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.717+0000 7fd5567fc700 1 --2- 172.21.15.27:0/3621683566 >> [v2:172.21.15.124:6824/90894,v1:172.21.15.124:6825/90894] conn(0x7fd54806f8b0 0x7fd548071cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.755 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.717+0000 7fd5567fc700 1 -- 172.21.15.27:0/3621683566 --> [v2:172.21.15.124:6824/90894,v1:172.21.15.124:6825/90894] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd548072370 con 0x7fd54806f8b0 2023-10-06T01:41:23.755 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.717+0000 7fd5567fc700 1 -- 172.21.15.27:0/3621683566 <== mon.1 v2:172.21.15.124:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd558085be0 con 0x7fd56815df30 2023-10-06T01:41:23.756 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.717+0000 7f52a7fff700 1 --2- 172.21.15.27:0/807096333 >> [v2:172.21.15.124:6800/90863,v1:172.21.15.124:6801/90863] conn(0x7f529406f760 0x7f5294071b70 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:23.756 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.717+0000 7fbdecdb8700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fbde8159fa0 0x7fbde815a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.756 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.717+0000 7fbdecdb8700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbde815a940 con 0x7fbde80cb800 2023-10-06T01:41:23.756 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.717+0000 7fbdecdb8700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbde81544a0 con 0x7fbde80af040 2023-10-06T01:41:23.756 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.717+0000 7fbdecdb8700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbde80b1600 con 0x7fbde8159fa0 2023-10-06T01:41:23.757 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.717+0000 7fd56659c700 1 --2- 172.21.15.27:0/3621683566 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd548067000 0x7fd5480694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.757 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.717+0000 7fd565d9b700 1 --2- 172.21.15.27:0/3621683566 >> [v2:172.21.15.124:6824/90894,v1:172.21.15.124:6825/90894] conn(0x7fd54806f8b0 0x7fd548071cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.757 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.717+0000 7fbde659c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fbde80cb800 0x7fbde80b1a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46270/0 (socket says 172.21.15.27:46270) 2023-10-06T01:41:23.757 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.717+0000 7fbde659c700 1 -- 172.21.15.27:0/610238445 learned_addr learned my addr 172.21.15.27:0/610238445 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:23.757 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.717+0000 7fbde5d9b700 1 --2- 172.21.15.27:0/610238445 >> v2:172.21.15.124:3300/0 conn(0x7fbde8159fa0 0x7fbde815a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.758 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.717+0000 7fd9037fe700 1 --2- 172.21.15.27:0/1482986497 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd908067080 0x7fd908069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.758 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.717+0000 7fd9037fe700 1 -- 172.21.15.27:0/1482986497 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fd9100853c0 con 0x7fd91c0cb980 2023-10-06T01:41:23.758 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.717+0000 7fd9037fe700 1 --2- 172.21.15.27:0/1482986497 >> [v2:172.21.15.27:6816/90645,v1:172.21.15.27:6817/90645] conn(0x7fd90806f8b0 0x7fd908071cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.758 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.717+0000 7fd9037fe700 1 -- 172.21.15.27:0/1482986497 --> [v2:172.21.15.27:6816/90645,v1:172.21.15.27:6817/90645] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd908072370 con 0x7fd90806f8b0 2023-10-06T01:41:23.758 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.717+0000 7f18ee686700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18e8068890 0x7f18e8145170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.758 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.717+0000 7f18ee686700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18e8068c60 0x7f18e81607a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.758 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.717+0000 7f18ee686700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f18e80afb90 0x7f18e8149ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.758 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.717+0000 7f18ee686700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e8161010 con 0x7f18e8068c60 2023-10-06T01:41:23.759 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.717+0000 7f18ee686700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e81612f0 con 0x7f18e8068890 2023-10-06T01:41:23.759 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.717+0000 7f18ee686700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18e81615d0 con 0x7f18e80afb90 2023-10-06T01:41:23.759 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.717+0000 7fd91bfff700 1 --2- 172.21.15.27:0/1482986497 >> [v2:172.21.15.27:6816/90645,v1:172.21.15.27:6817/90645] conn(0x7fd90806f8b0 0x7fd908071cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.759 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.718+0000 7fd91b7fe700 1 --2- 172.21.15.27:0/1482986497 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd908067080 0x7fd908069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.759 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.717+0000 7f18e7fff700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18e8068890 0x7f18e8145170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.759 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.717+0000 7f18e7fff700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18e8068890 0x7f18e8145170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.27:43126/0 (socket says 172.21.15.27:43126) 2023-10-06T01:41:23.759 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.717+0000 7f18e7fff700 1 -- 172.21.15.27:0/1610109108 learned_addr learned my addr 172.21.15.27:0/1610109108 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:23.760 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.718+0000 7fd9037fe700 1 -- 172.21.15.27:0/1482986497 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd910016e70 con 0x7fd91c0cb980 2023-10-06T01:41:23.760 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.718+0000 7f18e77fe700 1 --2- 172.21.15.27:0/1610109108 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18e8068c60 0x7f18e81607a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.760 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.718+0000 7fbde559a700 1 -- 172.21.15.27:0/610238445 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3405018621 0 0) 0x7fbddc001290 con 0x7fbde80cb800 2023-10-06T01:41:23.760 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.718+0000 7fbde559a700 1 -- 172.21.15.27:0/610238445 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdd0003570 con 0x7fbde80cb800 2023-10-06T01:41:23.761 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.718+0000 7f46e96ad700 1 Processor -- start 2023-10-06T01:41:23.761 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.717+0000 7fc70bfff700 1 -- 172.21.15.27:0/3831465769 <== mon.1 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc714006420 con 0x7fc72406b2e0 2023-10-06T01:41:23.761 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.718+0000 7fc70bfff700 1 --2- 172.21.15.27:0/3831465769 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc710067080 0x7fc710069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.761 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.718+0000 7fc70bfff700 1 -- 172.21.15.27:0/3831465769 <== mon.1 v2:172.21.15.124:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fc714083a70 con 0x7fc72406b2e0 2023-10-06T01:41:23.761 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.718+0000 7fc7237fe700 1 --2- 172.21.15.27:0/3831465769 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc710067080 0x7fc710069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.761 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.718+0000 7fd91bfff700 1 --2- 172.21.15.27:0/1482986497 >> [v2:172.21.15.27:6816/90645,v1:172.21.15.27:6817/90645] conn(0x7fd90806f8b0 0x7fd908071cc0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:23.762 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.718+0000 7f18e7fff700 1 -- 172.21.15.27:0/1610109108 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f18e80afb90 msgr2=0x7f18e8149ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.762 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.718+0000 7f18e7fff700 1 --2- 172.21.15.27:0/1610109108 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f18e80afb90 0x7f18e8149ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.762 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.718+0000 7f18e7fff700 1 -- 172.21.15.27:0/1610109108 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18e8068c60 msgr2=0x7f18e81607a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.762 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.718+0000 7f18e7fff700 1 --2- 172.21.15.27:0/1610109108 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18e8068c60 0x7f18e81607a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.762 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.718+0000 7f18e7fff700 1 -- 172.21.15.27:0/1610109108 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e0009ae0 con 0x7f18e8068890 2023-10-06T01:41:23.762 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.718+0000 7fbde5d9b700 1 -- 172.21.15.27:0/610238445 >> v1:172.21.15.124:6789/0 conn(0x7fbde80af040 legacy=0x7fbde80af4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.763 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.718+0000 7fbde5d9b700 1 -- 172.21.15.27:0/610238445 >> v1:172.21.15.27:6789/0 conn(0x7fbde80cb800 legacy=0x7fbde80b1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.763 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.718+0000 7fbde5d9b700 1 -- 172.21.15.27:0/610238445 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde806be00 con 0x7fbde8159fa0 2023-10-06T01:41:23.763 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.718+0000 7fd56659c700 1 --2- 172.21.15.27:0/3621683566 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd548067000 0x7fd5480694b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd560005ca0 tx=0x7fd560001620 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:23.763 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.718+0000 7fd565d9b700 1 --2- 172.21.15.27:0/3621683566 >> [v2:172.21.15.124:6824/90894,v1:172.21.15.124:6825/90894] conn(0x7fd54806f8b0 0x7fd548071cc0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:23.763 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.718+0000 7fc70bfff700 1 --2- 172.21.15.27:0/3831465769 >> [v2:172.21.15.27:6808/90643,v1:172.21.15.27:6809/90643] conn(0x7fc71006f8b0 0x7fc710071cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.763 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.718+0000 7fc70bfff700 1 -- 172.21.15.27:0/3831465769 --> [v2:172.21.15.27:6808/90643,v1:172.21.15.27:6809/90643] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc710072370 con 0x7fc71006f8b0 2023-10-06T01:41:23.763 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.718+0000 7fc70bfff700 1 -- 172.21.15.27:0/3831465769 <== mon.1 v2:172.21.15.124:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc714050be0 con 0x7fc72406b2e0 2023-10-06T01:41:23.764 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.718+0000 7f18e77fe700 1 --2- 172.21.15.27:0/1610109108 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18e8068c60 0x7f18e81607a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:41:23.764 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.718+0000 7fc728943700 1 --2- 172.21.15.27:0/3831465769 >> [v2:172.21.15.27:6808/90643,v1:172.21.15.27:6809/90643] conn(0x7fc71006f8b0 0x7fc710071cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.764 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.718+0000 7fd91b7fe700 1 --2- 172.21.15.27:0/1482986497 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd908067080 0x7fd908069530 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd90c0075c0 tx=0x7fd90c009780 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:23.764 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.718+0000 7f18e7fff700 1 --2- 172.21.15.27:0/1610109108 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18e8068890 0x7f18e8145170 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f18dc00b810 tx=0x7f18dc009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:23.765 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.719+0000 7fc7237fe700 1 --2- 172.21.15.27:0/3831465769 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc710067080 0x7fc710069530 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc724163200 tx=0x7fc718009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:23.765 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.719+0000 7fbde5d9b700 1 --2- 172.21.15.27:0/610238445 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbde8159fa0 0x7fbde815a400 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbdd80098e0 tx=0x7fbdd80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=277ef198e0538b64 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:23.765 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.719+0000 7fc728943700 1 --2- 172.21.15.27:0/3831465769 >> [v2:172.21.15.27:6808/90643,v1:172.21.15.27:6809/90643] conn(0x7fc71006f8b0 0x7fc710071cc0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:23.766 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.719+0000 7fd5567fc700 1 -- 172.21.15.27:0/3621683566 <== osd.7 v2:172.21.15.124:6824/90894 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fd548072370 con 0x7fd54806f8b0 2023-10-06T01:41:23.766 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.719+0000 7f528f7fe700 1 -- 172.21.15.27:0/807096333 <== osd.4 v2:172.21.15.124:6800/90863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f5294072220 con 0x7f529406f760 2023-10-06T01:41:23.766 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.719+0000 7f18cffff700 1 -- 172.21.15.27:0/1610109108 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18dc00f510 con 0x7f18e8068890 2023-10-06T01:41:23.766 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.719+0000 7f18cffff700 1 -- 172.21.15.27:0/1610109108 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18dc00f690 con 0x7f18e8068890 2023-10-06T01:41:23.766 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.719+0000 7f18cffff700 1 -- 172.21.15.27:0/1610109108 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18dc00fa50 con 0x7f18e8068890 2023-10-06T01:41:23.767 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.718+0000 7f46e96ad700 1 -- start start 2023-10-06T01:41:23.767 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.719+0000 7f46e96ad700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f46e415bc70 0x7f46e40afbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.768 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.719+0000 7f46e96ad700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f46e40695e0 0x7f46e415f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.768 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.719+0000 7f46e96ad700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46e415ecd0 con 0x7f46e40cc480 2023-10-06T01:41:23.768 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.719+0000 7f46e96ad700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46e40b0840 con 0x7f46e40695e0 2023-10-06T01:41:23.768 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.719+0000 7f46e96ad700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46e40cbbd0 con 0x7f46e415bc70 2023-10-06T01:41:23.768 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.719+0000 7f46e37fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f46e40695e0 0x7f46e415f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.768 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.719+0000 7f46e37fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f46e40695e0 0x7f46e415f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55974/0 (socket says 172.21.15.27:55974) 2023-10-06T01:41:23.768 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.719+0000 7f46e37fe700 1 -- 172.21.15.27:0/900169935 learned_addr learned my addr 172.21.15.27:0/900169935 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:23.768 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.720+0000 7f46e37fe700 1 -- 172.21.15.27:0/900169935 >> v2:172.21.15.124:3300/0 conn(0x7f46e415bc70 msgr2=0x7f46e40afbf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-06T01:41:23.768 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.720+0000 7f46e37fe700 1 --2- 172.21.15.27:0/900169935 >> v2:172.21.15.124:3300/0 conn(0x7f46e415bc70 0x7f46e40afbf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.768 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.720+0000 7f46e37fe700 1 -- 172.21.15.27:0/900169935 >> v1:172.21.15.27:6789/0 conn(0x7f46e40cc480 legacy=0x7f46e4068ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.768 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.720+0000 7f46e37fe700 1 -- 172.21.15.27:0/900169935 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46e40b0560 con 0x7f46e40695e0 2023-10-06T01:41:23.768 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.720+0000 7f46e37fe700 1 --2- 172.21.15.27:0/900169935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46e40695e0 0x7f46e415f9e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f46e406d700 tx=0x7f46d4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3524787d066befa server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:23.769 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.720+0000 7f46e1ffb700 1 -- 172.21.15.27:0/900169935 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d400b860 con 0x7f46e40695e0 2023-10-06T01:41:23.769 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.720+0000 7f46e1ffb700 1 -- 172.21.15.27:0/900169935 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46d400b9c0 con 0x7f46e40695e0 2023-10-06T01:41:23.769 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.720+0000 7f46e1ffb700 1 -- 172.21.15.27:0/900169935 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d400bd60 con 0x7f46e40695e0 2023-10-06T01:41:23.769 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.719+0000 7f4f177fe700 1 -- 172.21.15.27:0/556383606 <== mon.2 v2:172.21.15.124:3301/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4f2001ab50 con 0x7f4f300c9380 2023-10-06T01:41:23.769 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.720+0000 7f4f177fe700 1 --2- 172.21.15.27:0/556383606 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4f1c066f30 0x7f4f1c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.769 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.720+0000 7f4f177fe700 1 -- 172.21.15.27:0/556383606 <== mon.2 v2:172.21.15.124:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f4f20086c60 con 0x7f4f300c9380 2023-10-06T01:41:23.769 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.720+0000 7f4f2f7fe700 1 --2- 172.21.15.27:0/556383606 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4f1c066f30 0x7f4f1c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.770 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.720+0000 7f4f177fe700 1 --2- 172.21.15.27:0/556383606 >> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] conn(0x7f4f1c06f7e0 0x7f4f1c071bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.770 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.720+0000 7f4f177fe700 1 -- 172.21.15.27:0/556383606 --> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4f1c0722a0 con 0x7f4f1c06f7e0 2023-10-06T01:41:23.770 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.720+0000 7f4f177fe700 1 -- 172.21.15.27:0/556383606 <== mon.2 v2:172.21.15.124:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f20087030 con 0x7f4f300c9380 2023-10-06T01:41:23.770 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.720+0000 7fd9037fe700 1 -- 172.21.15.27:0/1482986497 <== osd.2 v2:172.21.15.27:6816/90645 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fd908072370 con 0x7fd90806f8b0 2023-10-06T01:41:23.770 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.720+0000 7f18ee686700 1 -- 172.21.15.27:0/1610109108 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18e81618b0 con 0x7f18e8068890 2023-10-06T01:41:23.770 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.720+0000 7f18ee686700 1 -- 172.21.15.27:0/1610109108 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18e814a420 con 0x7f18e8068890 2023-10-06T01:41:23.771 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.721+0000 7fc70bfff700 1 -- 172.21.15.27:0/3831465769 <== osd.1 v2:172.21.15.27:6808/90643 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fc710072370 con 0x7fc71006f8b0 2023-10-06T01:41:23.771 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.721+0000 7f4f2f7fe700 1 --2- 172.21.15.27:0/556383606 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4f1c066f30 0x7f4f1c0693e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4f24006fd0 tx=0x7f4f24000f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:23.771 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.721+0000 7f46e96ad700 1 -- 172.21.15.27:0/900169935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46e40695e0 msgr2=0x7f46e415f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.721+0000 7f46e96ad700 1 --2- 172.21.15.27:0/900169935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46e40695e0 0x7f46e415f9e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f46e406d700 tx=0x7f46d4009250 comp rx=0 tx=0).stop 2023-10-06T01:41:23.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.721+0000 7f46e96ad700 1 -- 172.21.15.27:0/900169935 shutdown_connections 2023-10-06T01:41:23.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.721+0000 7f46e96ad700 1 --2- 172.21.15.27:0/900169935 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46e40695e0 0x7f46e415f9e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.721+0000 7f46e96ad700 1 --2- 172.21.15.27:0/900169935 >> v2:172.21.15.124:3300/0 conn(0x7f46e415bc70 0x7f46e40afbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.721+0000 7f46e96ad700 1 -- 172.21.15.27:0/900169935 >> 172.21.15.27:0/900169935 conn(0x7f46e406e910 msgr2=0x7f46e406be30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:23.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.721+0000 7f46e96ad700 1 -- 172.21.15.27:0/900169935 shutdown_connections 2023-10-06T01:41:23.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.721+0000 7f18ee686700 1 -- 172.21.15.27:0/1610109108 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f18e8161460 con 0x7f18e8068890 2023-10-06T01:41:23.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.721+0000 7f46e96ad700 1 -- 172.21.15.27:0/900169935 wait complete. 2023-10-06T01:41:23.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.722+0000 7fbde559a700 1 -- 172.21.15.27:0/610238445 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd8011630 con 0x7fbde8159fa0 2023-10-06T01:41:23.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.722+0000 7fbde559a700 1 -- 172.21.15.27:0/610238445 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdd8011790 con 0x7fbde8159fa0 2023-10-06T01:41:23.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.722+0000 7fbde559a700 1 -- 172.21.15.27:0/610238445 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdd8011b30 con 0x7fbde8159fa0 2023-10-06T01:41:23.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.722+0000 7f18cffff700 1 -- 172.21.15.27:0/1610109108 <== mon.1 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f18dc01a400 con 0x7f18e8068890 2023-10-06T01:41:23.774 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.723+0000 7fbdecdb8700 1 -- 172.21.15.27:0/610238445 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbde8159fa0 msgr2=0x7fbde815a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.774 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.723+0000 7fbdecdb8700 1 --2- 172.21.15.27:0/610238445 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbde8159fa0 0x7fbde815a400 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbdd80098e0 tx=0x7fbdd80091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.774 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.723+0000 7f18cffff700 1 --2- 172.21.15.27:0/1610109108 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f18d4067000 0x7f18d40694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.774 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.723+0000 7f46e96ad700 1 Processor -- start 2023-10-06T01:41:23.775 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.723+0000 7f18cffff700 1 -- 172.21.15.27:0/1610109108 <== mon.1 v2:172.21.15.124:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f18dc084110 con 0x7f18e8068890 2023-10-06T01:41:23.775 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.723+0000 7fbdecdb8700 1 -- 172.21.15.27:0/610238445 shutdown_connections 2023-10-06T01:41:23.775 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.723+0000 7fbdecdb8700 1 --2- 172.21.15.27:0/610238445 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbde8159fa0 0x7fbde815a400 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.775 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.723+0000 7fbdecdb8700 1 -- 172.21.15.27:0/610238445 >> 172.21.15.27:0/610238445 conn(0x7fbde8068b90 msgr2=0x7fbde80694b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:23.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.723+0000 7f18cffff700 1 --2- 172.21.15.27:0/1610109108 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x7f18d406f8b0 0x7f18d4071cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.723+0000 7f18cffff700 1 -- 172.21.15.27:0/1610109108 --> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f18d4072370 con 0x7f18d406f8b0 2023-10-06T01:41:23.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.723+0000 7f18cffff700 1 -- 172.21.15.27:0/1610109108 <== mon.1 v2:172.21.15.124:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f18dc086740 con 0x7f18e8068890 2023-10-06T01:41:23.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.723+0000 7f18ecc23700 1 --2- 172.21.15.27:0/1610109108 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x7f18d406f8b0 0x7f18d4071cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.724+0000 7f18ecc23700 1 --2- 172.21.15.27:0/1610109108 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x7f18d406f8b0 0x7f18d4071cc0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:23.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.723+0000 7f46e96ad700 1 -- start start 2023-10-06T01:41:23.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.724+0000 7f46e96ad700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f46e40695e0 0x7f46e406b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.724+0000 7f46e96ad700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46e40cc480 0x7f46e40ae7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.724+0000 7f46e96ad700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f46e415bc70 0x7f46e414a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.724+0000 7f46e96ad700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e40ae360 con 0x7f46e40cc480 2023-10-06T01:41:23.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.724+0000 7f46e96ad700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e406b470 con 0x7f46e40695e0 2023-10-06T01:41:23.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.724+0000 7f46e96ad700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46e40aed30 con 0x7f46e415bc70 2023-10-06T01:41:23.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.724+0000 7f46e27fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46e40cc480 0x7f46e40ae7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.724+0000 7f46e27fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46e40cc480 0x7f46e40ae7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:55986/0 (socket says 172.21.15.27:55986) 2023-10-06T01:41:23.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.724+0000 7f46e27fc700 1 -- 172.21.15.27:0/4172749556 learned_addr learned my addr 172.21.15.27:0/4172749556 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:23.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.724+0000 7f46e37fe700 1 --2- 172.21.15.27:0/4172749556 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f46e415bc70 0x7f46e414a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.724+0000 7f46e27fc700 1 -- 172.21.15.27:0/4172749556 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f46e415bc70 msgr2=0x7f46e414a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.724+0000 7f46e27fc700 1 --2- 172.21.15.27:0/4172749556 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f46e415bc70 0x7f46e414a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.724+0000 7f46e27fc700 1 -- 172.21.15.27:0/4172749556 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f46e40695e0 msgr2=0x7f46e406b800 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-06T01:41:23.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.724+0000 7f46e27fc700 1 --2- 172.21.15.27:0/4172749556 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f46e40695e0 0x7f46e406b800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.724+0000 7f46e27fc700 1 -- 172.21.15.27:0/4172749556 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d40094f0 con 0x7f46e40cc480 2023-10-06T01:41:23.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.725+0000 7f18e77fe700 1 --2- 172.21.15.27:0/1610109108 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f18d4067000 0x7f18d40694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.779 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.725+0000 7f46e27fc700 1 --2- 172.21.15.27:0/4172749556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46e40cc480 0x7f46e40ae7f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f46d800c7c0 tx=0x7f46d8007f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:23.779 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.725+0000 7f46caffd700 1 -- 172.21.15.27:0/4172749556 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d8011590 con 0x7f46e40cc480 2023-10-06T01:41:23.779 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.725+0000 7f46caffd700 1 -- 172.21.15.27:0/4172749556 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46d80116f0 con 0x7f46e40cc480 2023-10-06T01:41:23.779 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.725+0000 7f46caffd700 1 -- 172.21.15.27:0/4172749556 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d8011ab0 con 0x7f46e40cc480 2023-10-06T01:41:23.780 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.725+0000 7f46e96ad700 1 -- 172.21.15.27:0/4172749556 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46e40ad750 con 0x7f46e40cc480 2023-10-06T01:41:23.780 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.725+0000 7f46e96ad700 1 -- 172.21.15.27:0/4172749556 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46e40add10 con 0x7f46e40cc480 2023-10-06T01:41:23.780 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.725+0000 7f18cffff700 1 -- 172.21.15.27:0/1610109108 <== osd.3 v2:172.21.15.27:6824/90668 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f18d4072370 con 0x7f18d406f8b0 2023-10-06T01:41:23.780 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.727+0000 7f46caffd700 1 -- 172.21.15.27:0/4172749556 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f46d801b3b0 con 0x7f46e40cc480 2023-10-06T01:41:23.780 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.727+0000 7f18e77fe700 1 --2- 172.21.15.27:0/1610109108 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f18d4067000 0x7f18d40694b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f18e814b070 tx=0x7f18d80090f0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:23.781 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.727+0000 7f46caffd700 1 --2- 172.21.15.27:0/4172749556 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f46d0066f30 0x7f46d00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.781 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.727+0000 7f46caffd700 1 -- 172.21.15.27:0/4172749556 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f46d8083be0 con 0x7f46e40cc480 2023-10-06T01:41:23.781 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.728+0000 7f46e2ffd700 1 --2- 172.21.15.27:0/4172749556 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f46d0066f30 0x7f46d00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.781 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.728+0000 7f46e2ffd700 1 --2- 172.21.15.27:0/4172749556 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f46d0066f30 0x7f46d00693e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f46dc00ad10 tx=0x7f46dc009370 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:23.782 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.729+0000 7f4f2ffff700 1 --2- 172.21.15.27:0/556383606 >> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] conn(0x7f4f1c06f7e0 0x7f4f1c071bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.782 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.729+0000 7f4f2ffff700 1 --2- 172.21.15.27:0/556383606 >> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] conn(0x7f4f1c06f7e0 0x7f4f1c071bf0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:23.782 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.730+0000 7f4f177fe700 1 -- 172.21.15.27:0/556383606 <== osd.0 v2:172.21.15.27:6800/90644 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f4f1c0722a0 con 0x7f4f1c06f7e0 2023-10-06T01:41:23.782 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.736+0000 7fbdecdb8700 1 -- 172.21.15.27:0/610238445 shutdown_connections 2023-10-06T01:41:23.782 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.736+0000 7fbdecdb8700 1 -- 172.21.15.27:0/610238445 wait complete. 2023-10-06T01:41:23.783 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.737+0000 7fd56cba1700 1 -- 172.21.15.27:0/3621683566 --> [v2:172.21.15.124:6824/90894,v1:172.21.15.124:6825/90894] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd5680cbf10 con 0x7fd54806f8b0 2023-10-06T01:41:23.783 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.738+0000 7fbdecdb8700 1 Processor -- start 2023-10-06T01:41:23.783 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.738+0000 7fbdecdb8700 1 -- start start 2023-10-06T01:41:23.783 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.738+0000 7fd5567fc700 1 -- 172.21.15.27:0/3621683566 <== osd.7 v2:172.21.15.124:6824/90894 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd5680cbf10 con 0x7fd54806f8b0 2023-10-06T01:41:23.784 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.739+0000 7fbdecdb8700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbde80af040 0x7fbde80998e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.784 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.739+0000 7fbdecdb8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbde80cb800 0x7fbde81522d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.784 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.739+0000 7fbdecdb8700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbde8159fa0 0x7fbde8150490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.784 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.739+0000 7fbdecdb8700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde80acb20 con 0x7fbde80cb800 2023-10-06T01:41:23.784 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.739+0000 7fbdecdb8700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde80abfa0 con 0x7fbde8159fa0 2023-10-06T01:41:23.785 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.739+0000 7fbdecdb8700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbde80ac280 con 0x7fbde80af040 2023-10-06T01:41:23.785 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.739+0000 7fbde5d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbde80cb800 0x7fbde81522d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.785 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.739+0000 7fbde6d9d700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbde8159fa0 0x7fbde8150490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.785 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.739+0000 7fbde5d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbde80cb800 0x7fbde81522d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56000/0 (socket says 172.21.15.27:56000) 2023-10-06T01:41:23.785 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.739+0000 7fbde6d9d700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbde8159fa0 0x7fbde8150490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.27:43156/0 (socket says 172.21.15.27:43156) 2023-10-06T01:41:23.785 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.739+0000 7fbde5d9b700 1 -- 172.21.15.27:0/2778774790 learned_addr learned my addr 172.21.15.27:0/2778774790 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:23.785 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.739+0000 7fd56cba1700 1 -- 172.21.15.27:0/3621683566 >> [v2:172.21.15.124:6824/90894,v1:172.21.15.124:6825/90894] conn(0x7fd54806f8b0 msgr2=0x7fd548071cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.786 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.739+0000 7fd56cba1700 1 --2- 172.21.15.27:0/3621683566 >> [v2:172.21.15.124:6824/90894,v1:172.21.15.124:6825/90894] conn(0x7fd54806f8b0 0x7fd548071cc0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.786 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.739+0000 7fd56cba1700 1 -- 172.21.15.27:0/3621683566 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd548067000 msgr2=0x7fd5480694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.786 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.739+0000 7fd56cba1700 1 --2- 172.21.15.27:0/3621683566 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd548067000 0x7fd5480694b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd560005ca0 tx=0x7fd560001620 comp rx=0 tx=0).stop 2023-10-06T01:41:23.786 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.739+0000 7fd56cba1700 1 -- 172.21.15.27:0/3621683566 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd56815df30 msgr2=0x7fd5680cae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.786 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.739+0000 7fd56cba1700 1 --2- 172.21.15.27:0/3621683566 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd56815df30 0x7fd5680cae10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd55800d890 tx=0x7fd558008040 comp rx=0 tx=0).stop 2023-10-06T01:41:23.786 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.739+0000 7fd566d9d700 1 -- 172.21.15.27:0/3621683566 reap_dead start 2023-10-06T01:41:23.788 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.739+0000 7fbde6d9d700 1 -- 172.21.15.27:0/2778774790 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbde80af040 msgr2=0x7fbde80998e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.788 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.739+0000 7fbde6d9d700 1 --2- 172.21.15.27:0/2778774790 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbde80af040 0x7fbde80998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.788 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.740+0000 7fbde6d9d700 1 -- 172.21.15.27:0/2778774790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbde80cb800 msgr2=0x7fbde81522d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.788 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.740+0000 7fbde6d9d700 1 --2- 172.21.15.27:0/2778774790 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbde80cb800 0x7fbde81522d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.788 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.740+0000 7fbde6d9d700 1 -- 172.21.15.27:0/2778774790 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd8009420 con 0x7fbde8159fa0 2023-10-06T01:41:23.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.740+0000 7fbde6d9d700 1 --2- 172.21.15.27:0/2778774790 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbde8159fa0 0x7fbde8150490 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbde0002230 tx=0x7fbde0001660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:23.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.740+0000 7fbdd67fc700 1 -- 172.21.15.27:0/2778774790 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde0012630 con 0x7fbde8159fa0 2023-10-06T01:41:23.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.740+0000 7fbdd67fc700 1 -- 172.21.15.27:0/2778774790 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde00127b0 con 0x7fbde8159fa0 2023-10-06T01:41:23.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.740+0000 7fbdd67fc700 1 -- 172.21.15.27:0/2778774790 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde0012b70 con 0x7fbde8159fa0 2023-10-06T01:41:23.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.741+0000 7fbdecdb8700 1 -- 172.21.15.27:0/2778774790 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbde80ac560 con 0x7fbde8159fa0 2023-10-06T01:41:23.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.741+0000 7fbdecdb8700 1 -- 172.21.15.27:0/2778774790 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbde8099e20 con 0x7fbde8159fa0 2023-10-06T01:41:23.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.741+0000 7fbdecdb8700 1 -- 172.21.15.27:0/2778774790 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbde8064050 con 0x7fbde8159fa0 2023-10-06T01:41:23.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.743+0000 7fbdd67fc700 1 -- 172.21.15.27:0/2778774790 <== mon.1 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fbde0003030 con 0x7fbde8159fa0 2023-10-06T01:41:23.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.743+0000 7fbdd67fc700 1 --2- 172.21.15.27:0/2778774790 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fbdc8066f30 0x7fbdc80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.743+0000 7fbdd67fc700 1 -- 172.21.15.27:0/2778774790 <== mon.1 v2:172.21.15.124:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fbde00844c0 con 0x7fbde8159fa0 2023-10-06T01:41:23.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.743+0000 7fbde659c700 1 --2- 172.21.15.27:0/2778774790 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fbdc8066f30 0x7fbdc80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.743+0000 7fbdd67fc700 1 --2- 172.21.15.27:0/2778774790 >> [v2:172.21.15.124:6808/90864,v1:172.21.15.124:6809/90864] conn(0x7fbdc806f7e0 0x7fbdc8071bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.743+0000 7fbdd67fc700 1 -- 172.21.15.27:0/2778774790 --> [v2:172.21.15.124:6808/90864,v1:172.21.15.124:6809/90864] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbdc80722a0 con 0x7fbdc806f7e0 2023-10-06T01:41:23.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.743+0000 7fbdd67fc700 1 -- 172.21.15.27:0/2778774790 <== mon.1 v2:172.21.15.124:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbde0086aa0 con 0x7fbde8159fa0 2023-10-06T01:41:23.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.743+0000 7fbde5d9b700 1 --2- 172.21.15.27:0/2778774790 >> [v2:172.21.15.124:6808/90864,v1:172.21.15.124:6809/90864] conn(0x7fbdc806f7e0 0x7fbdc8071bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.744+0000 7fbde659c700 1 --2- 172.21.15.27:0/2778774790 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fbdc8066f30 0x7fbdc80693e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbddc0014f0 tx=0x7fbddc001390 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:23.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.744+0000 7fbde5d9b700 1 --2- 172.21.15.27:0/2778774790 >> [v2:172.21.15.124:6808/90864,v1:172.21.15.124:6809/90864] conn(0x7fbdc806f7e0 0x7fbdc8071bf0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:23.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.745+0000 7fbdd67fc700 1 -- 172.21.15.27:0/2778774790 <== osd.6 v2:172.21.15.124:6808/90864 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fbdc80722a0 con 0x7fbdc806f7e0 2023-10-06T01:41:23.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.745+0000 7fd56cba1700 1 -- 172.21.15.27:0/3621683566 shutdown_connections 2023-10-06T01:41:23.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.745+0000 7fd56cba1700 1 -- 172.21.15.27:0/3621683566 >> 172.21.15.27:0/3621683566 conn(0x7fd568068b90 msgr2=0x7fd56815ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:23.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.748+0000 7fc72a3a6700 1 -- 172.21.15.27:0/3831465769 --> [v2:172.21.15.27:6808/90643,v1:172.21.15.27:6809/90643] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc72406bb60 con 0x7fc71006f8b0 2023-10-06T01:41:23.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.748+0000 7fc70bfff700 1 -- 172.21.15.27:0/3831465769 <== osd.1 v2:172.21.15.27:6808/90643 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc72406bb60 con 0x7fc71006f8b0 2023-10-06T01:41:23.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.748+0000 7f18ee686700 1 -- 172.21.15.27:0/1610109108 --> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f18e8161a40 con 0x7f18d406f8b0 2023-10-06T01:41:23.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.749+0000 7f18cffff700 1 -- 172.21.15.27:0/1610109108 <== osd.3 v2:172.21.15.27:6824/90668 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f18e8161a40 con 0x7f18d406f8b0 2023-10-06T01:41:23.792 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.749+0000 7fc72a3a6700 1 -- 172.21.15.27:0/3831465769 >> [v2:172.21.15.27:6808/90643,v1:172.21.15.27:6809/90643] conn(0x7fc71006f8b0 msgr2=0x7fc710071cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.792 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.749+0000 7fc72a3a6700 1 --2- 172.21.15.27:0/3831465769 >> [v2:172.21.15.27:6808/90643,v1:172.21.15.27:6809/90643] conn(0x7fc71006f8b0 0x7fc710071cc0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.792 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.749+0000 7fc72a3a6700 1 -- 172.21.15.27:0/3831465769 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc710067080 msgr2=0x7fc710069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.793 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.749+0000 7fc72a3a6700 1 --2- 172.21.15.27:0/3831465769 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc710067080 0x7fc710069530 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc724163200 tx=0x7fc718009040 comp rx=0 tx=0).stop 2023-10-06T01:41:23.793 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.749+0000 7fc72a3a6700 1 -- 172.21.15.27:0/3831465769 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc72406b2e0 msgr2=0x7fc7240a6280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.793 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.749+0000 7fc72a3a6700 1 --2- 172.21.15.27:0/3831465769 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc72406b2e0 0x7fc7240a6280 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc71400b530 tx=0x7fc7140090f0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.793 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.749+0000 7fc728943700 1 -- 172.21.15.27:0/3831465769 reap_dead start 2023-10-06T01:41:23.793 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.749+0000 7f46e96ad700 1 --2- 172.21.15.27:0/4172749556 >> [v2:172.21.15.124:6816/90885,v1:172.21.15.124:6817/90885] conn(0x7f46e40adf00 0x7f46e415f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:23.793 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.749+0000 7f46e96ad700 1 -- 172.21.15.27:0/4172749556 --> [v2:172.21.15.124:6816/90885,v1:172.21.15.124:6817/90885] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f46e4067b80 con 0x7f46e40adf00 2023-10-06T01:41:23.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.749+0000 7fc72a3a6700 1 -- 172.21.15.27:0/3831465769 shutdown_connections 2023-10-06T01:41:23.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.749+0000 7fc72a3a6700 1 -- 172.21.15.27:0/3831465769 >> 172.21.15.27:0/3831465769 conn(0x7fc724068b90 msgr2=0x7fc7240cadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:23.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.749+0000 7fc72a3a6700 1 -- 172.21.15.27:0/3831465769 shutdown_connections 2023-10-06T01:41:23.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.749+0000 7fc72a3a6700 1 -- 172.21.15.27:0/3831465769 wait complete. 2023-10-06T01:41:23.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.749+0000 7f46e37fe700 1 --2- 172.21.15.27:0/4172749556 >> [v2:172.21.15.124:6816/90885,v1:172.21.15.124:6817/90885] conn(0x7f46e40adf00 0x7f46e415f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:23.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.750+0000 7f46e37fe700 1 --2- 172.21.15.27:0/4172749556 >> [v2:172.21.15.124:6816/90885,v1:172.21.15.124:6817/90885] conn(0x7f46e40adf00 0x7f46e415f9e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:23.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.750+0000 7f46caffd700 1 -- 172.21.15.27:0/4172749556 <== osd.5 v2:172.21.15.124:6816/90885 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f46e4067b80 con 0x7f46e40adf00 2023-10-06T01:41:23.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.751+0000 7fd56cba1700 1 -- 172.21.15.27:0/3621683566 shutdown_connections 2023-10-06T01:41:23.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.749+0000 7f52ae1b1700 1 -- 172.21.15.27:0/807096333 --> [v2:172.21.15.124:6800/90863,v1:172.21.15.124:6801/90863] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f52a8097040 con 0x7f529406f760 2023-10-06T01:41:23.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.749+0000 7f528f7fe700 1 -- 172.21.15.27:0/807096333 <== osd.4 v2:172.21.15.124:6800/90863 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f52a8097040 con 0x7f529406f760 2023-10-06T01:41:23.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.750+0000 7f52ae1b1700 1 -- 172.21.15.27:0/807096333 >> [v2:172.21.15.124:6800/90863,v1:172.21.15.124:6801/90863] conn(0x7f529406f760 msgr2=0x7f5294071b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.750+0000 7f52ae1b1700 1 --2- 172.21.15.27:0/807096333 >> [v2:172.21.15.124:6800/90863,v1:172.21.15.124:6801/90863] conn(0x7f529406f760 0x7f5294071b70 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.750+0000 7f52ae1b1700 1 -- 172.21.15.27:0/807096333 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5294066f30 msgr2=0x7f52940693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.750+0000 7f52ae1b1700 1 --2- 172.21.15.27:0/807096333 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5294066f30 0x7f52940693e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5298009fa0 tx=0x7f529800b040 comp rx=0 tx=0).stop 2023-10-06T01:41:23.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.750+0000 7f52ae1b1700 1 -- 172.21.15.27:0/807096333 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f52a80c7500 msgr2=0x7f52a817f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.750+0000 7f52ae1b1700 1 --2- 172.21.15.27:0/807096333 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f52a80c7500 0x7f52a817f470 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f529c014000 tx=0x7f529c015000 comp rx=0 tx=0).stop 2023-10-06T01:41:23.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.750+0000 7f52a7fff700 1 -- 172.21.15.27:0/807096333 reap_dead start 2023-10-06T01:41:23.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.750+0000 7f52ae1b1700 1 -- 172.21.15.27:0/807096333 shutdown_connections 2023-10-06T01:41:23.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.750+0000 7f52ae1b1700 1 -- 172.21.15.27:0/807096333 >> 172.21.15.27:0/807096333 conn(0x7f52a8068b90 msgr2=0x7f52a80ad830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:23.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.750+0000 7f52ae1b1700 1 -- 172.21.15.27:0/807096333 shutdown_connections 2023-10-06T01:41:23.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.750+0000 7f52ae1b1700 1 -- 172.21.15.27:0/807096333 wait complete. 2023-10-06T01:41:23.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.756+0000 7fd56cba1700 1 -- 172.21.15.27:0/3621683566 wait complete. 2023-10-06T01:41:23.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.756+0000 7f4f35cb6700 1 -- 172.21.15.27:0/556383606 --> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4f30064050 con 0x7f4f1c06f7e0 2023-10-06T01:41:23.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.756+0000 7fd921dd6700 1 -- 172.21.15.27:0/1482986497 --> [v2:172.21.15.27:6816/90645,v1:172.21.15.27:6817/90645] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd91c0ad920 con 0x7fd90806f8b0 2023-10-06T01:41:23.798 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.756+0000 7f18ee686700 1 -- 172.21.15.27:0/1610109108 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x7f18d406f8b0 msgr2=0x7f18d4071cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.798 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.756+0000 7f18ee686700 1 --2- 172.21.15.27:0/1610109108 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x7f18d406f8b0 0x7f18d4071cc0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.798 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.756+0000 7f4f177fe700 1 -- 172.21.15.27:0/556383606 <== osd.0 v2:172.21.15.27:6800/90644 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4f30064050 con 0x7f4f1c06f7e0 2023-10-06T01:41:23.798 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.757+0000 7fd9037fe700 1 -- 172.21.15.27:0/1482986497 <== osd.2 v2:172.21.15.27:6816/90645 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd91c0ad920 con 0x7fd90806f8b0 2023-10-06T01:41:23.799 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.757+0000 7f4f35cb6700 1 -- 172.21.15.27:0/556383606 >> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] conn(0x7f4f1c06f7e0 msgr2=0x7f4f1c071bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.799 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.757+0000 7f4f35cb6700 1 --2- 172.21.15.27:0/556383606 >> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] conn(0x7f4f1c06f7e0 0x7f4f1c071bf0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.799 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.757+0000 7f4f35cb6700 1 -- 172.21.15.27:0/556383606 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4f1c066f30 msgr2=0x7f4f1c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.800 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.757+0000 7f4f35cb6700 1 --2- 172.21.15.27:0/556383606 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4f1c066f30 0x7f4f1c0693e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4f24006fd0 tx=0x7f4f24000f90 comp rx=0 tx=0).stop 2023-10-06T01:41:23.800 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.757+0000 7f4f35cb6700 1 -- 172.21.15.27:0/556383606 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4f300c9380 msgr2=0x7f4f300b0b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.800 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.757+0000 7f4f35cb6700 1 --2- 172.21.15.27:0/556383606 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4f300c9380 0x7f4f300b0b10 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4f30063350 tx=0x7f4f20017000 comp rx=0 tx=0).stop 2023-10-06T01:41:23.800 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.757+0000 7f4f2ffff700 1 -- 172.21.15.27:0/556383606 reap_dead start 2023-10-06T01:41:23.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.757+0000 7fd921dd6700 1 -- 172.21.15.27:0/1482986497 >> [v2:172.21.15.27:6816/90645,v1:172.21.15.27:6817/90645] conn(0x7fd90806f8b0 msgr2=0x7fd908071cc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.757+0000 7fd921dd6700 1 --2- 172.21.15.27:0/1482986497 >> [v2:172.21.15.27:6816/90645,v1:172.21.15.27:6817/90645] conn(0x7fd90806f8b0 0x7fd908071cc0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.757+0000 7fd921dd6700 1 -- 172.21.15.27:0/1482986497 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd908067080 msgr2=0x7fd908069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.757+0000 7fd921dd6700 1 --2- 172.21.15.27:0/1482986497 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd908067080 0x7fd908069530 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd90c0075c0 tx=0x7fd90c009780 comp rx=0 tx=0).stop 2023-10-06T01:41:23.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.757+0000 7fd921dd6700 1 -- 172.21.15.27:0/1482986497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd91c0cb980 msgr2=0x7fd91c0a8a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.757+0000 7fd921dd6700 1 --2- 172.21.15.27:0/1482986497 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd91c0cb980 0x7fd91c0a8a80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd910008880 tx=0x7fd91001b000 comp rx=0 tx=0).stop 2023-10-06T01:41:23.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.757+0000 7fd91bfff700 1 -- 172.21.15.27:0/1482986497 reap_dead start 2023-10-06T01:41:23.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.760+0000 7fd921dd6700 1 -- 172.21.15.27:0/1482986497 shutdown_connections 2023-10-06T01:41:23.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.760+0000 7fd921dd6700 1 -- 172.21.15.27:0/1482986497 >> 172.21.15.27:0/1482986497 conn(0x7fd91c068b90 msgr2=0x7fd91c0b1850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:23.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.760+0000 7f18ee686700 1 -- 172.21.15.27:0/1610109108 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f18d4067000 msgr2=0x7f18d40694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.760+0000 7f18ee686700 1 --2- 172.21.15.27:0/1610109108 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f18d4067000 0x7f18d40694b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f18e814b070 tx=0x7f18d80090f0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.760+0000 7f18ee686700 1 -- 172.21.15.27:0/1610109108 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18e8068890 msgr2=0x7f18e8145170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.760+0000 7f18ee686700 1 --2- 172.21.15.27:0/1610109108 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18e8068890 0x7f18e8145170 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f18dc00b810 tx=0x7f18dc009250 comp rx=0 tx=0).stop 2023-10-06T01:41:23.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.760+0000 7f18ecc23700 1 -- 172.21.15.27:0/1610109108 reap_dead start 2023-10-06T01:41:23.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.760+0000 7f18ee686700 1 -- 172.21.15.27:0/1610109108 shutdown_connections 2023-10-06T01:41:23.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.760+0000 7f18ee686700 1 -- 172.21.15.27:0/1610109108 >> 172.21.15.27:0/1610109108 conn(0x7f18e806be70 msgr2=0x7f18e80cbc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:23.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.761+0000 7f4f35cb6700 1 -- 172.21.15.27:0/556383606 shutdown_connections 2023-10-06T01:41:23.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.761+0000 7f4f35cb6700 1 -- 172.21.15.27:0/556383606 >> 172.21.15.27:0/556383606 conn(0x7f4f3006e5f0 msgr2=0x7f4f30156c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:23.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.762+0000 7fd921dd6700 1 -- 172.21.15.27:0/1482986497 shutdown_connections 2023-10-06T01:41:23.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.762+0000 7f18ee686700 1 -- 172.21.15.27:0/1610109108 shutdown_connections 2023-10-06T01:41:23.804 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.762+0000 7fd921dd6700 1 -- 172.21.15.27:0/1482986497 wait complete. 2023-10-06T01:41:23.804 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.762+0000 7f18ee686700 1 -- 172.21.15.27:0/1610109108 wait complete. 2023-10-06T01:41:23.805 INFO:teuthology.orchestra.run.smithi027.stdout:55834574855 2023-10-06T01:41:23.805 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-06T01:41:23.805 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.765+0000 7f4f35cb6700 1 -- 172.21.15.27:0/556383606 shutdown_connections 2023-10-06T01:41:23.806 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.765+0000 7f4f35cb6700 1 -- 172.21.15.27:0/556383606 wait complete. 2023-10-06T01:41:23.806 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.768+0000 7fbdecdb8700 1 -- 172.21.15.27:0/2778774790 --> [v2:172.21.15.124:6808/90864,v1:172.21.15.124:6809/90864] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbde81599e0 con 0x7fbdc806f7e0 2023-10-06T01:41:23.806 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.768+0000 7fbdd67fc700 1 -- 172.21.15.27:0/2778774790 <== osd.6 v2:172.21.15.124:6808/90864 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fbde81599e0 con 0x7fbdc806f7e0 2023-10-06T01:41:23.807 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.769+0000 7fbdecdb8700 1 -- 172.21.15.27:0/2778774790 >> [v2:172.21.15.124:6808/90864,v1:172.21.15.124:6809/90864] conn(0x7fbdc806f7e0 msgr2=0x7fbdc8071bf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.807 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.769+0000 7fbdecdb8700 1 --2- 172.21.15.27:0/2778774790 >> [v2:172.21.15.124:6808/90864,v1:172.21.15.124:6809/90864] conn(0x7fbdc806f7e0 0x7fbdc8071bf0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.807 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.769+0000 7fbdecdb8700 1 -- 172.21.15.27:0/2778774790 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fbdc8066f30 msgr2=0x7fbdc80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.807 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.769+0000 7fbdecdb8700 1 --2- 172.21.15.27:0/2778774790 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fbdc8066f30 0x7fbdc80693e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbddc0014f0 tx=0x7fbddc001390 comp rx=0 tx=0).stop 2023-10-06T01:41:23.807 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.769+0000 7fbdecdb8700 1 -- 172.21.15.27:0/2778774790 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbde8159fa0 msgr2=0x7fbde8150490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.808 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.769+0000 7fbdecdb8700 1 --2- 172.21.15.27:0/2778774790 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbde8159fa0 0x7fbde8150490 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbde0002230 tx=0x7fbde0001660 comp rx=0 tx=0).stop 2023-10-06T01:41:23.808 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.769+0000 7fbde6d9d700 1 -- 172.21.15.27:0/2778774790 reap_dead start 2023-10-06T01:41:23.808 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.770+0000 7fbdecdb8700 1 -- 172.21.15.27:0/2778774790 shutdown_connections 2023-10-06T01:41:23.808 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.770+0000 7fbdecdb8700 1 -- 172.21.15.27:0/2778774790 >> 172.21.15.27:0/2778774790 conn(0x7fbde8068b90 msgr2=0x7fbde8153760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:23.808 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.770+0000 7fbdecdb8700 1 -- 172.21.15.27:0/2778774790 shutdown_connections 2023-10-06T01:41:23.808 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.770+0000 7fbdecdb8700 1 -- 172.21.15.27:0/2778774790 wait complete. 2023-10-06T01:41:23.809 INFO:teuthology.orchestra.run.smithi027.stdout:55834574856 2023-10-06T01:41:23.809 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-06T01:41:23.809 INFO:teuthology.orchestra.run.smithi027.stdout:55834574857 2023-10-06T01:41:23.810 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-06T01:41:23.810 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.777+0000 7f46e96ad700 1 -- 172.21.15.27:0/4172749556 --> [v2:172.21.15.124:6816/90885,v1:172.21.15.124:6817/90885] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f46e4063880 con 0x7f46e40adf00 2023-10-06T01:41:23.810 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.777+0000 7f46caffd700 1 -- 172.21.15.27:0/4172749556 <== osd.5 v2:172.21.15.124:6816/90885 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f46e4063880 con 0x7f46e40adf00 2023-10-06T01:41:23.811 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.778+0000 7f46e96ad700 1 -- 172.21.15.27:0/4172749556 >> [v2:172.21.15.124:6816/90885,v1:172.21.15.124:6817/90885] conn(0x7f46e40adf00 msgr2=0x7f46e415f9e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.812 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.778+0000 7f46e96ad700 1 --2- 172.21.15.27:0/4172749556 >> [v2:172.21.15.124:6816/90885,v1:172.21.15.124:6817/90885] conn(0x7f46e40adf00 0x7f46e415f9e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:23.812 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.778+0000 7f46e96ad700 1 -- 172.21.15.27:0/4172749556 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f46d0066f30 msgr2=0x7f46d00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.812 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.778+0000 7f46e96ad700 1 --2- 172.21.15.27:0/4172749556 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f46d0066f30 0x7f46d00693e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f46dc00ad10 tx=0x7f46dc009370 comp rx=0 tx=0).stop 2023-10-06T01:41:23.812 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.778+0000 7f46e96ad700 1 -- 172.21.15.27:0/4172749556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46e40cc480 msgr2=0x7f46e40ae7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:23.812 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.778+0000 7f46e96ad700 1 --2- 172.21.15.27:0/4172749556 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f46e40cc480 0x7f46e40ae7f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f46d800c7c0 tx=0x7f46d8007f50 comp rx=0 tx=0).stop 2023-10-06T01:41:23.812 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.778+0000 7f46e37fe700 1 -- 172.21.15.27:0/4172749556 reap_dead start 2023-10-06T01:41:23.812 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.778+0000 7f46e96ad700 1 -- 172.21.15.27:0/4172749556 shutdown_connections 2023-10-06T01:41:23.812 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.778+0000 7f46e96ad700 1 -- 172.21.15.27:0/4172749556 >> 172.21.15.27:0/4172749556 conn(0x7f46e406e910 msgr2=0x7f46e415d800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:23.812 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.783+0000 7f46e96ad700 1 -- 172.21.15.27:0/4172749556 shutdown_connections 2023-10-06T01:41:23.812 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:23.783+0000 7f46e96ad700 1 -- 172.21.15.27:0/4172749556 wait complete. 2023-10-06T01:41:23.813 INFO:teuthology.orchestra.run.smithi027.stdout:55834574855 2023-10-06T01:41:23.813 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-06T01:41:23.814 INFO:teuthology.orchestra.run.smithi027.stdout:55834574855 2023-10-06T01:41:23.814 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-06T01:41:23.815 INFO:teuthology.orchestra.run.smithi027.stdout:55834574857 2023-10-06T01:41:23.815 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-06T01:41:23.816 INFO:teuthology.orchestra.run.smithi027.stdout:55834574856 2023-10-06T01:41:23.816 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-06T01:41:23.817 INFO:teuthology.orchestra.run.smithi027.stdout:55834574854 2023-10-06T01:41:23.817 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-06T01:41:24.046 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.045+0000 7f882e95b700 1 Processor -- start 2023-10-06T01:41:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.053+0000 7f882e95b700 1 -- start start 2023-10-06T01:41:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.055+0000 7f882e95b700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f88280cb980 0x7f88280ae400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.055+0000 7f882e95b700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f8828156930 0x7f88280b15d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.055+0000 7f882e95b700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88280adf70 con 0x7f88280ab3b0 2023-10-06T01:41:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.055+0000 7f882e95b700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88280b0e00 con 0x7f88280cb980 2023-10-06T01:41:24.055 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.055+0000 7f882e95b700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88280ad360 con 0x7f8828156930 2023-10-06T01:41:24.056 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.055+0000 7f882cef8700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f8828156930 0x7f88280b15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.056 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.055+0000 7f882cef8700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f8828156930 0x7f88280b15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.27:35984/0 (socket says 172.21.15.27:35984) 2023-10-06T01:41:24.056 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.055+0000 7f882cef8700 1 -- 172.21.15.27:0/893175504 learned_addr learned my addr 172.21.15.27:0/893175504 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:24.056 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.055+0000 7f882cef8700 1 -- 172.21.15.27:0/893175504 >> v2:172.21.15.124:3300/0 conn(0x7f88280cb980 msgr2=0x7f88280ae400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:24.056 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.055+0000 7f882cef8700 1 --2- 172.21.15.27:0/893175504 >> v2:172.21.15.124:3300/0 conn(0x7f88280cb980 0x7f88280ae400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.057 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.055+0000 7f882cef8700 1 -- 172.21.15.27:0/893175504 >> v1:172.21.15.27:6789/0 conn(0x7f88280ab3b0 legacy=0x7f8828156220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:24.057 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.055+0000 7f882cef8700 1 -- 172.21.15.27:0/893175504 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88280ad640 con 0x7f8828156930 2023-10-06T01:41:24.057 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.056+0000 7f882cef8700 1 --2- 172.21.15.27:0/893175504 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f8828156930 0x7f88280b15d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f881c0089b0 tx=0x7f881c0082d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5055dac027c2d480 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:24.057 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.056+0000 7f8826ffd700 1 -- 172.21.15.27:0/893175504 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f881c010070 con 0x7f8828156930 2023-10-06T01:41:24.057 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.056+0000 7f8826ffd700 1 -- 172.21.15.27:0/893175504 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f881c00daf0 con 0x7f8828156930 2023-10-06T01:41:24.057 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.056+0000 7f8826ffd700 1 -- 172.21.15.27:0/893175504 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f881c00de90 con 0x7f8828156930 2023-10-06T01:41:24.058 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.057+0000 7f882e95b700 1 -- 172.21.15.27:0/893175504 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f8828156930 msgr2=0x7f88280b15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.058 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.057+0000 7f882e95b700 1 --2- 172.21.15.27:0/893175504 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f8828156930 0x7f88280b15d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f881c0089b0 tx=0x7f881c0082d0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.058 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.057+0000 7f882e95b700 1 -- 172.21.15.27:0/893175504 shutdown_connections 2023-10-06T01:41:24.058 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.057+0000 7f882e95b700 1 --2- 172.21.15.27:0/893175504 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f8828156930 0x7f88280b15d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.058 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.057+0000 7f882e95b700 1 --2- 172.21.15.27:0/893175504 >> v2:172.21.15.124:3300/0 conn(0x7f88280cb980 0x7f88280ae400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.058 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.057+0000 7f882e95b700 1 -- 172.21.15.27:0/893175504 >> 172.21.15.27:0/893175504 conn(0x7f8828068b90 msgr2=0x7f88280694b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:24.058 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.057+0000 7f882e95b700 1 -- 172.21.15.27:0/893175504 shutdown_connections 2023-10-06T01:41:24.058 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.057+0000 7f882e95b700 1 -- 172.21.15.27:0/893175504 wait complete. 2023-10-06T01:41:24.060 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.060+0000 7f882e95b700 1 Processor -- start 2023-10-06T01:41:24.060 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.060+0000 7f882e95b700 1 -- start start 2023-10-06T01:41:24.063 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.061+0000 7f882e95b700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f88280ab3b0 0x7f8828169cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.063 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.061+0000 7f882e95b700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f88280cb980 0x7f88280a73a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.063 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.061+0000 7f882e95b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8828156930 0x7f88280b15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.064 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.061+0000 7f882e95b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88280c5540 con 0x7f8828156930 2023-10-06T01:41:24.064 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.061+0000 7f882e95b700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88280a8e80 con 0x7f88280ab3b0 2023-10-06T01:41:24.064 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.061+0000 7f882e95b700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88280a9160 con 0x7f88280cb980 2023-10-06T01:41:24.064 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.061+0000 7f8827fff700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f88280ab3b0 0x7f8828169cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.064 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.061+0000 7f88277fe700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f88280cb980 0x7f88280a73a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.064 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.061+0000 7f8827fff700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f88280ab3b0 0x7f8828169cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.27:43160/0 (socket says 172.21.15.27:43160) 2023-10-06T01:41:24.064 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.061+0000 7f8827fff700 1 -- 172.21.15.27:0/1403943215 learned_addr learned my addr 172.21.15.27:0/1403943215 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:24.064 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.062+0000 7f8827fff700 1 -- 172.21.15.27:0/1403943215 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f88280cb980 msgr2=0x7f88280a73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.064 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.062+0000 7f8827fff700 1 --2- 172.21.15.27:0/1403943215 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f88280cb980 0x7f88280a73a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.065 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.062+0000 7f8827fff700 1 -- 172.21.15.27:0/1403943215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8828156930 msgr2=0x7f88280b15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.065 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.062+0000 7f8827fff700 1 --2- 172.21.15.27:0/1403943215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8828156930 0x7f88280b15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.065 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.062+0000 7f8827fff700 1 -- 172.21.15.27:0/1403943215 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f881c008570 con 0x7f88280ab3b0 2023-10-06T01:41:24.065 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.062+0000 7f8827fff700 1 --2- 172.21.15.27:0/1403943215 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f88280ab3b0 0x7f8828169cf0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f882000bc90 tx=0x7f8820009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:24.065 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.062+0000 7f880ffff700 1 -- 172.21.15.27:0/1403943215 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88200092e0 con 0x7f88280ab3b0 2023-10-06T01:41:24.065 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.062+0000 7f880ffff700 1 -- 172.21.15.27:0/1403943215 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f882000c750 con 0x7f88280ab3b0 2023-10-06T01:41:24.065 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.062+0000 7f880ffff700 1 -- 172.21.15.27:0/1403943215 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882000cb10 con 0x7f88280ab3b0 2023-10-06T01:41:24.065 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.062+0000 7f882e95b700 1 -- 172.21.15.27:0/1403943215 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88280a7cf0 con 0x7f88280ab3b0 2023-10-06T01:41:24.065 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.062+0000 7f882e95b700 1 -- 172.21.15.27:0/1403943215 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88280a82b0 con 0x7f88280ab3b0 2023-10-06T01:41:24.066 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.064+0000 7f882e95b700 1 -- 172.21.15.27:0/1403943215 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88080052b0 con 0x7f88280ab3b0 2023-10-06T01:41:24.066 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.064+0000 7fdb8527a700 1 Processor -- start 2023-10-06T01:41:24.066 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.064+0000 7f2b70511700 1 Processor -- start 2023-10-06T01:41:24.066 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.064+0000 7f2b70511700 1 -- start start 2023-10-06T01:41:24.067 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.064+0000 7f880ffff700 1 -- 172.21.15.27:0/1403943215 <== mon.1 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8820012070 con 0x7f88280ab3b0 2023-10-06T01:41:24.067 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.065+0000 7f880ffff700 1 --2- 172.21.15.27:0/1403943215 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8814066f30 0x7f88140693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.067 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.065+0000 7f880ffff700 1 -- 172.21.15.27:0/1403943215 <== mon.1 v2:172.21.15.124:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f88200841c0 con 0x7f88280ab3b0 2023-10-06T01:41:24.067 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.065+0000 7ff1900cf700 1 Processor -- start 2023-10-06T01:41:24.068 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.065+0000 7fed8d91b700 1 Processor -- start 2023-10-06T01:41:24.068 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.064+0000 7fdb8527a700 1 -- start start 2023-10-06T01:41:24.068 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.065+0000 7fdb8527a700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdb8015d920 0x7fdb800b3de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.068 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.065+0000 7fdb8527a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdb8015dcf0 0x7fdb800cd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.068 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.065+0000 7fdb8527a700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb800aefe0 con 0x7fdb800cdde0 2023-10-06T01:41:24.068 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.065+0000 7fdb8527a700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb800ce260 con 0x7fdb8015dcf0 2023-10-06T01:41:24.068 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.065+0000 7fdb8527a700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb800b4320 con 0x7fdb8015d920 2023-10-06T01:41:24.069 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.066+0000 7f2b70511700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2b680c80d0 0x7f2b681613a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.069 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.066+0000 7f2b70511700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b680c87a0 con 0x7f2b6813fb10 2023-10-06T01:41:24.069 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.066+0000 7f2b70511700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b6806adb0 con 0x7f2b6813fee0 2023-10-06T01:41:24.069 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.066+0000 7f2b70511700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b6806b090 con 0x7f2b680c80d0 2023-10-06T01:41:24.069 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.066+0000 7f88277fe700 1 --2- 172.21.15.27:0/1403943215 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8814066f30 0x7f88140693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.070 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.066+0000 7fdb7effd700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdb8015d920 0x7fdb800b3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.070 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.066+0000 7fdb7effd700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdb8015d920 0x7fdb800b3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.27:35992/0 (socket says 172.21.15.27:35992) 2023-10-06T01:41:24.070 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.066+0000 7fdb7effd700 1 -- 172.21.15.27:0/1138645399 learned_addr learned my addr 172.21.15.27:0/1138645399 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:24.070 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.065+0000 7ff1900cf700 1 -- start start 2023-10-06T01:41:24.070 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.066+0000 7f2b6e2ad700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2b6813fb10 0x7f2b6806a5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46288/0 (socket says 172.21.15.27:46288) 2023-10-06T01:41:24.071 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.066+0000 7f2b6e2ad700 1 -- 172.21.15.27:0/345213413 learned_addr learned my addr 172.21.15.27:0/345213413 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:24.071 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.066+0000 7fdb7e7fc700 1 --2- 172.21.15.27:0/1138645399 >> v2:172.21.15.27:3300/0 conn(0x7fdb8015dcf0 0x7fdb800cd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.071 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.067+0000 7ff1900cf700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff18815d890 0x7ff18815dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.071 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.067+0000 7ff1900cf700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff1880b4b90 0x7ff1880694c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.071 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.067+0000 7ff1900cf700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1880cc900 con 0x7ff1880b41f0 2023-10-06T01:41:24.071 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.067+0000 7ff1900cf700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1880cd260 con 0x7ff18815d890 2023-10-06T01:41:24.071 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.067+0000 7ff1900cf700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1880d0520 con 0x7ff1880b4b90 2023-10-06T01:41:24.072 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.067+0000 7fdb7e7fc700 1 -- 172.21.15.27:0/1138645399 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdb8015d920 msgr2=0x7fdb800b3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:24.072 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.067+0000 7fdb7e7fc700 1 --2- 172.21.15.27:0/1138645399 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdb8015d920 0x7fdb800b3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.072 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.067+0000 7fdb7e7fc700 1 -- 172.21.15.27:0/1138645399 >> v1:172.21.15.124:6789/0 conn(0x7fdb800cdde0 legacy=0x7fdb801616f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-06T01:41:24.072 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.067+0000 7fdb7e7fc700 1 -- 172.21.15.27:0/1138645399 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb800af230 con 0x7fdb8015dcf0 2023-10-06T01:41:24.072 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.067+0000 7ff18de6b700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff18815d890 0x7ff18815dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.072 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.067+0000 7ff18e66c700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff1880b4b90 0x7ff1880694c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.072 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.067+0000 7ff18de6b700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff18815d890 0x7ff18815dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.27:43174/0 (socket says 172.21.15.27:43174) 2023-10-06T01:41:24.073 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.067+0000 7ff18de6b700 1 -- 172.21.15.27:0/2728488301 learned_addr learned my addr 172.21.15.27:0/2728488301 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:24.073 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.067+0000 7f2b6d2ab700 1 -- 172.21.15.27:0/345213413 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1929647043 0 0) 0x7f2b680c87a0 con 0x7f2b6813fb10 2023-10-06T01:41:24.073 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.067+0000 7f2b6d2ab700 1 -- 172.21.15.27:0/345213413 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b54003570 con 0x7f2b6813fb10 2023-10-06T01:41:24.073 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.066+0000 7fed8d91b700 1 -- start start 2023-10-06T01:41:24.074 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.067+0000 7fed8d91b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fed88069120 0x7fed880ade90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.074 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.067+0000 7fed8d91b700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fed880694f0 0x7fed88153600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.074 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.067+0000 7fed8d91b700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed880ac3e0 con 0x7fed88156550 2023-10-06T01:41:24.074 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.067+0000 7fed8d91b700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed880b0380 con 0x7fed88069120 2023-10-06T01:41:24.074 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.067+0000 7fed8d91b700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed88156ac0 con 0x7fed880694f0 2023-10-06T01:41:24.074 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.067+0000 7fdb7e7fc700 1 --2- 172.21.15.27:0/1138645399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb8015dcf0 0x7fdb800cd8a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdb740098e0 tx=0x7fdb740091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95ff6a2d0f2bdbd server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:24.075 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.067+0000 7f2b6d2ab700 1 -- 172.21.15.27:0/345213413 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2732035977 0 0) 0x7f2b54003570 con 0x7f2b6813fb10 2023-10-06T01:41:24.075 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.067+0000 7f2b6d2ab700 1 -- 172.21.15.27:0/345213413 >> v2:172.21.15.27:3300/0 conn(0x7f2b680c80d0 msgr2=0x7f2b681613a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-06T01:41:24.075 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.067+0000 7f2b6d2ab700 1 --2- 172.21.15.27:0/345213413 >> v2:172.21.15.27:3300/0 conn(0x7f2b680c80d0 0x7f2b681613a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.075 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.067+0000 7f2b6d2ab700 1 -- 172.21.15.27:0/345213413 >> v1:172.21.15.124:6789/0 conn(0x7f2b6813fee0 legacy=0x7f2b680c79c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-06T01:41:24.075 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.067+0000 7f2b6d2ab700 1 -- 172.21.15.27:0/345213413 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b6806b370 con 0x7f2b6813fb10 2023-10-06T01:41:24.076 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.067+0000 7f2b6d2ab700 1 -- 172.21.15.27:0/345213413 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2619078944 0 0) 0x7f2b64001470 con 0x7f2b6813fb10 2023-10-06T01:41:24.076 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.068+0000 7ff18de6b700 1 -- 172.21.15.27:0/2728488301 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff1880b4b90 msgr2=0x7ff1880694c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:24.076 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.068+0000 7ff18de6b700 1 --2- 172.21.15.27:0/2728488301 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff1880b4b90 0x7ff1880694c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.077 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.068+0000 7ff18de6b700 1 -- 172.21.15.27:0/2728488301 >> v1:172.21.15.124:6789/0 conn(0x7ff1880b41f0 legacy=0x7ff1880cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.077 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.068+0000 7ff18de6b700 1 -- 172.21.15.27:0/2728488301 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1880ccba0 con 0x7ff18815d890 2023-10-06T01:41:24.078 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.068+0000 7f2b6d2ab700 1 -- 172.21.15.27:0/345213413 >> v1:172.21.15.27:6789/0 conn(0x7f2b6813fb10 legacy=0x7f2b6806a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.078 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.068+0000 7f2b6d2ab700 1 --2- 172.21.15.27:0/345213413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b54003ae0 0x7f2b54003ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.078 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.068+0000 7f2b6d2ab700 1 --2- 172.21.15.27:0/345213413 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2b54004810 0x7f2b54004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.078 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.068+0000 7f2b6d2ab700 1 --2- 172.21.15.27:0/345213413 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2b540053c0 0x7f2b5400ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.078 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.068+0000 7f2b6d2ab700 1 -- 172.21.15.27:0/345213413 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b6806b090 con 0x7f2b54003ae0 2023-10-06T01:41:24.079 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.068+0000 7f2b6d2ab700 1 -- 172.21.15.27:0/345213413 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b54014b90 con 0x7f2b54004810 2023-10-06T01:41:24.079 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.068+0000 7f2b6d2ab700 1 -- 172.21.15.27:0/345213413 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b54014d50 con 0x7f2b540053c0 2023-10-06T01:41:24.079 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.068+0000 7ff18de6b700 1 --2- 172.21.15.27:0/2728488301 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff18815d890 0x7ff18815dc60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff1780099e0 tx=0x7ff178009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4712282fd5d13bba server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:24.080 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.068+0000 7fdb7dffb700 1 -- 172.21.15.27:0/1138645399 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb74009e80 con 0x7fdb8015dcf0 2023-10-06T01:41:24.080 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.068+0000 7fdb7dffb700 1 -- 172.21.15.27:0/1138645399 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb7400b820 con 0x7fdb8015dcf0 2023-10-06T01:41:24.080 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.068+0000 7fdb7dffb700 1 -- 172.21.15.27:0/1138645399 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb7400bba0 con 0x7fdb8015dcf0 2023-10-06T01:41:24.080 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.068+0000 7f2b6e2ad700 1 --2- 172.21.15.27:0/345213413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b54003ae0 0x7f2b54003ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.081 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.068+0000 7fed877fe700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fed880694f0 0x7fed88153600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.081 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.068+0000 7fed877fe700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fed880694f0 0x7fed88153600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.27:36004/0 (socket says 172.21.15.27:36004) 2023-10-06T01:41:24.081 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.068+0000 7fed877fe700 1 -- 172.21.15.27:0/2837892073 learned_addr learned my addr 172.21.15.27:0/2837892073 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:24.082 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.068+0000 7fed867fc700 1 --1- 172.21.15.27:0/2837892073 >> v1:172.21.15.27:6789/0 conn(0x7fed88156550 0x7fed880afc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:46300/0 (socket says 172.21.15.27:46300) 2023-10-06T01:41:24.082 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.068+0000 7fed86ffd700 1 --2- 172.21.15.27:0/2837892073 >> v2:172.21.15.27:3300/0 conn(0x7fed88069120 0x7fed880ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.082 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.068+0000 7fed877fe700 1 -- 172.21.15.27:0/2837892073 >> v2:172.21.15.27:3300/0 conn(0x7fed88069120 msgr2=0x7fed880ade90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:24.082 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.068+0000 7fed877fe700 1 --2- 172.21.15.27:0/2837892073 >> v2:172.21.15.27:3300/0 conn(0x7fed88069120 0x7fed880ade90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.082 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.068+0000 7fed877fe700 1 -- 172.21.15.27:0/2837892073 >> v1:172.21.15.27:6789/0 conn(0x7fed88156550 legacy=0x7fed880afc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:24.082 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.068+0000 7fed877fe700 1 -- 172.21.15.27:0/2837892073 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed88156da0 con 0x7fed880694f0 2023-10-06T01:41:24.082 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.068+0000 7fed877fe700 1 --2- 172.21.15.27:0/2837892073 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fed880694f0 0x7fed88153600 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fed780099b0 tx=0x7fed78009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aedd534b4d9c76e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:24.083 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.069+0000 7fed85ffb700 1 -- 172.21.15.27:0/2837892073 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7800b820 con 0x7fed880694f0 2023-10-06T01:41:24.083 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.069+0000 7f2b6eaae700 1 --2- 172.21.15.27:0/345213413 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2b540053c0 0x7f2b5400ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.083 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.069+0000 7f2b6e2ad700 1 -- 172.21.15.27:0/345213413 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2b540053c0 msgr2=0x7f2b5400ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:24.084 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.069+0000 7f2b6e2ad700 1 --2- 172.21.15.27:0/345213413 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2b540053c0 0x7f2b5400ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.084 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.069+0000 7f2b6e2ad700 1 -- 172.21.15.27:0/345213413 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2b54004810 msgr2=0x7f2b54004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-06T01:41:24.084 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.069+0000 7f2b6e2ad700 1 --2- 172.21.15.27:0/345213413 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2b54004810 0x7f2b54004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.084 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.069+0000 7f2b6e2ad700 1 -- 172.21.15.27:0/345213413 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b6806b370 con 0x7f2b54003ae0 2023-10-06T01:41:24.084 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.069+0000 7f2b6eaae700 1 --2- 172.21.15.27:0/345213413 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2b540053c0 0x7f2b5400ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:41:24.084 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.069+0000 7f2b6eaae700 1 -- 172.21.15.27:0/345213413 reap_dead start 2023-10-06T01:41:24.084 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.069+0000 7fed85ffb700 1 -- 172.21.15.27:0/2837892073 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed7800b980 con 0x7fed880694f0 2023-10-06T01:41:24.084 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.069+0000 7fed85ffb700 1 -- 172.21.15.27:0/2837892073 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7800bd00 con 0x7fed880694f0 2023-10-06T01:41:24.085 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.069+0000 7f2b6e2ad700 1 --2- 172.21.15.27:0/345213413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b54003ae0 0x7f2b54003ed0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2b6400c320 tx=0x7f2b6400c630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba2f5de70ec125f1 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:24.085 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.069+0000 7f2b6d2ab700 1 -- 172.21.15.27:0/345213413 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b6400ccc0 con 0x7f2b54003ae0 2023-10-06T01:41:24.085 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.069+0000 7f2b6d2ab700 1 -- 172.21.15.27:0/345213413 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b6400ce20 con 0x7f2b54003ae0 2023-10-06T01:41:24.085 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.069+0000 7f2b6d2ab700 1 -- 172.21.15.27:0/345213413 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b64005b50 con 0x7f2b54003ae0 2023-10-06T01:41:24.086 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.070+0000 7fed8d91b700 1 -- 172.21.15.27:0/2837892073 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fed880694f0 msgr2=0x7fed88153600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.086 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.070+0000 7fed8d91b700 1 --2- 172.21.15.27:0/2837892073 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fed880694f0 0x7fed88153600 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fed780099b0 tx=0x7fed78009250 comp rx=0 tx=0).stop 2023-10-06T01:41:24.086 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.069+0000 7fdb8527a700 1 -- 172.21.15.27:0/1138645399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb8015dcf0 msgr2=0x7fdb800cd8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.086 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.069+0000 7fdb8527a700 1 --2- 172.21.15.27:0/1138645399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb8015dcf0 0x7fdb800cd8a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdb740098e0 tx=0x7fdb740091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.087 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.069+0000 7fdb8527a700 1 -- 172.21.15.27:0/1138645399 shutdown_connections 2023-10-06T01:41:24.087 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.069+0000 7fdb8527a700 1 --2- 172.21.15.27:0/1138645399 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdb8015d920 0x7fdb800b3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.087 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.069+0000 7fdb8527a700 1 --2- 172.21.15.27:0/1138645399 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb8015dcf0 0x7fdb800cd8a0 secure :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdb740098e0 tx=0x7fdb740091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.087 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.069+0000 7fdb8527a700 1 -- 172.21.15.27:0/1138645399 >> 172.21.15.27:0/1138645399 conn(0x7fdb8006d890 msgr2=0x7fdb8006dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:24.087 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.069+0000 7fdb8527a700 1 -- 172.21.15.27:0/1138645399 shutdown_connections 2023-10-06T01:41:24.087 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.069+0000 7fdb8527a700 1 -- 172.21.15.27:0/1138645399 wait complete. 2023-10-06T01:41:24.088 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.070+0000 7fed8d91b700 1 -- 172.21.15.27:0/2837892073 shutdown_connections 2023-10-06T01:41:24.088 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.070+0000 7fed8d91b700 1 --2- 172.21.15.27:0/2837892073 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fed880694f0 0x7fed88153600 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.088 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.070+0000 7fed8d91b700 1 --2- 172.21.15.27:0/2837892073 >> v2:172.21.15.27:3300/0 conn(0x7fed88069120 0x7fed880ade90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.088 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.070+0000 7fed8d91b700 1 -- 172.21.15.27:0/2837892073 >> 172.21.15.27:0/2837892073 conn(0x7fed8806e910 msgr2=0x7fed8806be30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:24.088 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.070+0000 7fed8d91b700 1 -- 172.21.15.27:0/2837892073 shutdown_connections 2023-10-06T01:41:24.089 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.070+0000 7fed8d91b700 1 -- 172.21.15.27:0/2837892073 wait complete. 2023-10-06T01:41:24.089 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.071+0000 7f29939a9700 1 Processor -- start 2023-10-06T01:41:24.089 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.070+0000 7ff18ce69700 1 -- 172.21.15.27:0/2728488301 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17800b820 con 0x7ff18815d890 2023-10-06T01:41:24.089 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.070+0000 7ff18ce69700 1 -- 172.21.15.27:0/2728488301 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff17800b980 con 0x7ff18815d890 2023-10-06T01:41:24.089 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.070+0000 7ff18ce69700 1 -- 172.21.15.27:0/2728488301 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff17800bd20 con 0x7ff18815d890 2023-10-06T01:41:24.090 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.071+0000 7ff1900cf700 1 -- 172.21.15.27:0/2728488301 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff18815d890 msgr2=0x7ff18815dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.090 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.071+0000 7ff1900cf700 1 --2- 172.21.15.27:0/2728488301 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff18815d890 0x7ff18815dc60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff1780099e0 tx=0x7ff178009250 comp rx=0 tx=0).stop 2023-10-06T01:41:24.090 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.070+0000 7f2b70511700 1 -- 172.21.15.27:0/345213413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b54003ae0 msgr2=0x7f2b54003ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.090 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.070+0000 7f2b70511700 1 --2- 172.21.15.27:0/345213413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b54003ae0 0x7f2b54003ed0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2b6400c320 tx=0x7f2b6400c630 comp rx=0 tx=0).stop 2023-10-06T01:41:24.091 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.071+0000 7f880ffff700 1 -- 172.21.15.27:0/1403943215 <== mon.1 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f88200512d0 con 0x7f88280ab3b0 2023-10-06T01:41:24.091 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.071+0000 7fdb8527a700 1 Processor -- start 2023-10-06T01:41:24.091 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.071+0000 7f88277fe700 1 --2- 172.21.15.27:0/1403943215 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8814066f30 0x7f88140693e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f882806f800 tx=0x7f8818008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:24.092 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.071+0000 7f29939a9700 1 -- start start 2023-10-06T01:41:24.092 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.072+0000 7f0d96bd5700 1 Processor -- start 2023-10-06T01:41:24.093 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.072+0000 7ff1900cf700 1 -- 172.21.15.27:0/2728488301 shutdown_connections 2023-10-06T01:41:24.093 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.072+0000 7ff1900cf700 1 --2- 172.21.15.27:0/2728488301 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff1880b4b90 0x7ff1880694c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.093 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.072+0000 7ff1900cf700 1 --2- 172.21.15.27:0/2728488301 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff18815d890 0x7ff18815dc60 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.093 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.072+0000 7ff1900cf700 1 -- 172.21.15.27:0/2728488301 >> 172.21.15.27:0/2728488301 conn(0x7ff1880680b0 msgr2=0x7ff1880b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:24.094 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.072+0000 7f0d96bd5700 1 -- start start 2023-10-06T01:41:24.094 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.072+0000 7fed8d91b700 1 Processor -- start 2023-10-06T01:41:24.095 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.072+0000 7fed8d91b700 1 -- start start 2023-10-06T01:41:24.095 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.072+0000 7fdb8527a700 1 -- start start 2023-10-06T01:41:24.095 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.073+0000 7fdb8527a700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdb800cdde0 0x7fdb80142e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.095 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.073+0000 7fdb8527a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb8015d920 0x7fdb8014ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.096 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.073+0000 7fdb8527a700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdb801433d0 0x7fdb8014efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.096 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.073+0000 7fdb8527a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8014b240 con 0x7fdb8015d920 2023-10-06T01:41:24.096 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.073+0000 7fdb8527a700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb800adb20 con 0x7fdb801433d0 2023-10-06T01:41:24.096 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.073+0000 7fdb8527a700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb8014a370 con 0x7fdb800cdde0 2023-10-06T01:41:24.097 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.073+0000 7f29939a9700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f298c0a69e0 0x7f298c0cb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.097 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.073+0000 7f29939a9700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f298c0aca20 0x7f298c0b1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.097 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.073+0000 7f29939a9700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f298c0add40 con 0x7f298c0a6db0 2023-10-06T01:41:24.097 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.073+0000 7f29939a9700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f298c0ae8e0 con 0x7f298c0a69e0 2023-10-06T01:41:24.097 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.073+0000 7f29939a9700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f298c15b2e0 con 0x7f298c0aca20 2023-10-06T01:41:24.098 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.073+0000 7f0d96bd5700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0d90159500 0x7f0d901598d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.098 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.073+0000 7f0d96bd5700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f0d900b3fd0 0x7f0d9015d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.098 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.073+0000 7f0d96bd5700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d9015dd20 con 0x7f0d9015d8e0 2023-10-06T01:41:24.099 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.073+0000 7f0d96bd5700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d900b1110 con 0x7f0d900b3fd0 2023-10-06T01:41:24.099 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.073+0000 7f0d96bd5700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d900b13b0 con 0x7f0d90159500 2023-10-06T01:41:24.099 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.073+0000 7fed8d91b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed88069120 0x7fed880cb7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.099 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.073+0000 7fed8d91b700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fed880694f0 0x7fed8806c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.100 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.073+0000 7fed8d91b700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fed88156550 0x7fed88152440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.100 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.073+0000 7fed8d91b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed881722e0 con 0x7fed88069120 2023-10-06T01:41:24.100 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.073+0000 7fed8d91b700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed881716d0 con 0x7fed880694f0 2023-10-06T01:41:24.100 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.073+0000 7fed8d91b700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed881719b0 con 0x7fed88156550 2023-10-06T01:41:24.101 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.072+0000 7ff1900cf700 1 -- 172.21.15.27:0/2728488301 shutdown_connections 2023-10-06T01:41:24.101 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.073+0000 7ff1900cf700 1 -- 172.21.15.27:0/2728488301 wait complete. 2023-10-06T01:41:24.101 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.073+0000 7f0d95172700 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f0d9015d8e0 0x7f0d900cf6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.27:46220/0 (socket says 172.21.15.27:46220) 2023-10-06T01:41:24.102 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.073+0000 7f0d95172700 1 -- 172.21.15.27:0/1658799220 learned_addr learned my addr 172.21.15.27:0/1658799220 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:24.102 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.073+0000 7fdb7f7fe700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdb801433d0 0x7fdb8014efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.102 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.074+0000 7fdb7f7fe700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdb801433d0 0x7fdb8014efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.27:43194/0 (socket says 172.21.15.27:43194) 2023-10-06T01:41:24.103 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.074+0000 7fdb7f7fe700 1 -- 172.21.15.27:0/45599546 learned_addr learned my addr 172.21.15.27:0/45599546 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:24.103 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.074+0000 7f0d94971700 1 --2- 172.21.15.27:0/1658799220 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0d90159500 0x7f0d901598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.104 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.073+0000 7fed867fc700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fed880694f0 0x7fed8806c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.104 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.073+0000 7fed877fe700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fed88156550 0x7fed88152440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.104 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.074+0000 7fed877fe700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fed88156550 0x7fed88152440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.27:36058/0 (socket says 172.21.15.27:36058) 2023-10-06T01:41:24.104 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.074+0000 7fed877fe700 1 -- 172.21.15.27:0/1168158451 learned_addr learned my addr 172.21.15.27:0/1168158451 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:24.105 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.073+0000 7f2991f46700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f298c0aca20 0x7f298c0b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.105 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.073+0000 7f2991f46700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f298c0aca20 0x7f298c0b1350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.27:36034/0 (socket says 172.21.15.27:36034) 2023-10-06T01:41:24.105 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.073+0000 7f2991f46700 1 -- 172.21.15.27:0/1614128667 learned_addr learned my addr 172.21.15.27:0/1614128667 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:24.106 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.074+0000 7f2991745700 1 --2- 172.21.15.27:0/1614128667 >> v2:172.21.15.124:3300/0 conn(0x7f298c0a69e0 0x7f298c0cb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.106 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.074+0000 7fdb7effd700 1 --2- 172.21.15.27:0/45599546 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdb800cdde0 0x7fdb80142e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.107 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.074+0000 7f0d8f7fe700 1 -- 172.21.15.27:0/1658799220 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 749042845 0 0) 0x7f0d9015dd20 con 0x7f0d9015d8e0 2023-10-06T01:41:24.107 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.074+0000 7f0d8f7fe700 1 -- 172.21.15.27:0/1658799220 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d78003570 con 0x7f0d9015d8e0 2023-10-06T01:41:24.107 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.074+0000 7fed877fe700 1 -- 172.21.15.27:0/1168158451 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fed880694f0 msgr2=0x7fed8806c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.108 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.074+0000 7fed877fe700 1 --2- 172.21.15.27:0/1168158451 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fed880694f0 0x7fed8806c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.108 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.074+0000 7fed877fe700 1 -- 172.21.15.27:0/1168158451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed88069120 msgr2=0x7fed880cb7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.108 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.074+0000 7fed877fe700 1 --2- 172.21.15.27:0/1168158451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed88069120 0x7fed880cb7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.108 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.074+0000 7fed877fe700 1 -- 172.21.15.27:0/1168158451 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed780094f0 con 0x7fed88156550 2023-10-06T01:41:24.109 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.074+0000 7f0d94971700 1 -- 172.21.15.27:0/1658799220 >> v2:172.21.15.27:3300/0 conn(0x7f0d900b3fd0 msgr2=0x7f0d9015d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:24.109 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.074+0000 7f0d94971700 1 --2- 172.21.15.27:0/1658799220 >> v2:172.21.15.27:3300/0 conn(0x7f0d900b3fd0 0x7f0d9015d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.109 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.074+0000 7f0d94971700 1 -- 172.21.15.27:0/1658799220 >> v1:172.21.15.124:6789/0 conn(0x7f0d9015d8e0 legacy=0x7f0d900cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.109 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.074+0000 7f0d94971700 1 -- 172.21.15.27:0/1658799220 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d900b1690 con 0x7f0d90159500 2023-10-06T01:41:24.110 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.074+0000 7f2991f46700 1 -- 172.21.15.27:0/1614128667 >> v2:172.21.15.124:3300/0 conn(0x7f298c0a69e0 msgr2=0x7f298c0cb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:24.110 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.074+0000 7f2991f46700 1 --2- 172.21.15.27:0/1614128667 >> v2:172.21.15.124:3300/0 conn(0x7f298c0a69e0 0x7f298c0cb6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.110 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.074+0000 7f2991f46700 1 -- 172.21.15.27:0/1614128667 >> v1:172.21.15.124:6789/0 conn(0x7f298c0a6db0 legacy=0x7f298c0ae1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.110 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.074+0000 7f2991f46700 1 -- 172.21.15.27:0/1614128667 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f298c0ad410 con 0x7f298c0aca20 2023-10-06T01:41:24.111 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.074+0000 7fed877fe700 1 --2- 172.21.15.27:0/1168158451 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fed88156550 0x7fed88152440 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fed78009290 tx=0x7fed78016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:24.111 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.074+0000 7f0d94971700 1 --2- 172.21.15.27:0/1658799220 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0d90159500 0x7f0d901598d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0d800098e0 tx=0x7f0d800091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ad56e6af9340fb3 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:24.112 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.074+0000 7ff1900cf700 1 Processor -- start 2023-10-06T01:41:24.112 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.074+0000 7fdb7effd700 1 -- 172.21.15.27:0/45599546 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdb801433d0 msgr2=0x7fdb8014efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.112 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.074+0000 7fdb7effd700 1 --2- 172.21.15.27:0/45599546 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdb801433d0 0x7fdb8014efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.112 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.074+0000 7fdb7effd700 1 -- 172.21.15.27:0/45599546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb8015d920 msgr2=0x7fdb8014ea80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-06T01:41:24.113 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.074+0000 7fdb7effd700 1 --2- 172.21.15.27:0/45599546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb8015d920 0x7fdb8014ea80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.113 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.074+0000 7fdb7effd700 1 -- 172.21.15.27:0/45599546 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb74009420 con 0x7fdb800cdde0 2023-10-06T01:41:24.113 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.074+0000 7f2991f46700 1 --2- 172.21.15.27:0/1614128667 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f298c0aca20 0x7f298c0b1350 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2984009f20 tx=0x7f2984009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c990b58cd0bebf5b server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:24.114 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.074+0000 7ff1900cf700 1 -- start start 2023-10-06T01:41:24.114 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.074+0000 7fed6effd700 1 -- 172.21.15.27:0/1168158451 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7801d070 con 0x7fed88156550 2023-10-06T01:41:24.114 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.075+0000 7fed6effd700 1 -- 172.21.15.27:0/1168158451 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed78023420 con 0x7fed88156550 2023-10-06T01:41:24.115 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.075+0000 7f2983fff700 1 -- 172.21.15.27:0/1614128667 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f298400e070 con 0x7f298c0aca20 2023-10-06T01:41:24.115 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.075+0000 7f2983fff700 1 -- 172.21.15.27:0/1614128667 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f298400b9d0 con 0x7f298c0aca20 2023-10-06T01:41:24.115 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.075+0000 7f0d8f7fe700 1 -- 172.21.15.27:0/1658799220 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d8000b640 con 0x7f0d90159500 2023-10-06T01:41:24.115 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.075+0000 7f2983fff700 1 -- 172.21.15.27:0/1614128667 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29840092e0 con 0x7f298c0aca20 2023-10-06T01:41:24.116 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.075+0000 7f0d8f7fe700 1 -- 172.21.15.27:0/1658799220 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d8000b7a0 con 0x7f0d90159500 2023-10-06T01:41:24.116 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.075+0000 7f0d8f7fe700 1 -- 172.21.15.27:0/1658799220 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d8000bb40 con 0x7f0d90159500 2023-10-06T01:41:24.116 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.075+0000 7ff1900cf700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff1880b41f0 0x7ff18815bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.116 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.075+0000 7ff1900cf700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff1880b4b90 0x7ff1880af2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.116 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.075+0000 7ff1900cf700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff18815d890 0x7ff18806cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.117 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.075+0000 7ff1900cf700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18806b1f0 con 0x7ff18815d890 2023-10-06T01:41:24.117 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.075+0000 7ff1900cf700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18806a5e0 con 0x7ff1880b4b90 2023-10-06T01:41:24.117 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.075+0000 7ff1900cf700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff18806a8c0 con 0x7ff1880b41f0 2023-10-06T01:41:24.117 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.075+0000 7fed6effd700 1 -- 172.21.15.27:0/1168158451 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7800b820 con 0x7fed88156550 2023-10-06T01:41:24.117 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.075+0000 7fed8d91b700 1 -- 172.21.15.27:0/1168158451 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed88171c90 con 0x7fed88156550 2023-10-06T01:41:24.117 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.075+0000 7f29939a9700 1 -- 172.21.15.27:0/1614128667 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f298c0aca20 msgr2=0x7f298c0b1350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.117 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.075+0000 7f29939a9700 1 --2- 172.21.15.27:0/1614128667 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f298c0aca20 0x7f298c0b1350 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2984009f20 tx=0x7f2984009840 comp rx=0 tx=0).stop 2023-10-06T01:41:24.118 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.076+0000 7ff18e66c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff18815d890 0x7ff18806cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.118 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.076+0000 7ff18de6b700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff1880b41f0 0x7ff18815bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.118 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.076+0000 7ff18e66c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff18815d890 0x7ff18806cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56046/0 (socket says 172.21.15.27:56046) 2023-10-06T01:41:24.118 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.076+0000 7ff18e66c700 1 -- 172.21.15.27:0/3158573190 learned_addr learned my addr 172.21.15.27:0/3158573190 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:24.118 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.076+0000 7fed8d91b700 1 -- 172.21.15.27:0/1168158451 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed880cbd30 con 0x7fed88156550 2023-10-06T01:41:24.119 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.076+0000 7ff18d66a700 1 --2- 172.21.15.27:0/3158573190 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff1880b4b90 0x7ff1880af2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.119 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.076+0000 7f0d96bd5700 1 -- 172.21.15.27:0/1658799220 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0d90159500 msgr2=0x7f0d901598d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.119 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.076+0000 7f0d96bd5700 1 --2- 172.21.15.27:0/1658799220 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0d90159500 0x7f0d901598d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0d800098e0 tx=0x7f0d800091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.119 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.076+0000 7f29939a9700 1 -- 172.21.15.27:0/1614128667 shutdown_connections 2023-10-06T01:41:24.119 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.076+0000 7f29939a9700 1 --2- 172.21.15.27:0/1614128667 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f298c0aca20 0x7f298c0b1350 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.120 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.076+0000 7f29939a9700 1 --2- 172.21.15.27:0/1614128667 >> v2:172.21.15.124:3300/0 conn(0x7f298c0a69e0 0x7f298c0cb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.120 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.076+0000 7f29939a9700 1 -- 172.21.15.27:0/1614128667 >> 172.21.15.27:0/1614128667 conn(0x7f298c068b90 msgr2=0x7f298c0694b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:24.120 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.076+0000 7f29939a9700 1 -- 172.21.15.27:0/1614128667 shutdown_connections 2023-10-06T01:41:24.120 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.076+0000 7f2b70511700 1 -- 172.21.15.27:0/345213413 shutdown_connections 2023-10-06T01:41:24.120 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.076+0000 7f2b70511700 1 --2- 172.21.15.27:0/345213413 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b54003ae0 0x7f2b54003ed0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.120 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.076+0000 7f2b70511700 1 -- 172.21.15.27:0/345213413 >> 172.21.15.27:0/345213413 conn(0x7f2b6806e760 msgr2=0x7f2b6806eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:24.121 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.076+0000 7f29939a9700 1 -- 172.21.15.27:0/1614128667 wait complete. 2023-10-06T01:41:24.121 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.076+0000 7ff18de6b700 1 -- 172.21.15.27:0/3158573190 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff1880b4b90 msgr2=0x7ff1880af2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.076+0000 7ff18de6b700 1 --2- 172.21.15.27:0/3158573190 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff1880b4b90 0x7ff1880af2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.076+0000 7ff18de6b700 1 -- 172.21.15.27:0/3158573190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff18815d890 msgr2=0x7ff18806cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.076+0000 7ff18de6b700 1 --2- 172.21.15.27:0/3158573190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff18815d890 0x7ff18806cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.076+0000 7ff18de6b700 1 -- 172.21.15.27:0/3158573190 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1780094f0 con 0x7ff1880b41f0 2023-10-06T01:41:24.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.076+0000 7ff18de6b700 1 --2- 172.21.15.27:0/3158573190 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff1880b41f0 0x7ff18815bed0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff1880afce0 tx=0x7ff178017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:24.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.076+0000 7ff17dffb700 1 -- 172.21.15.27:0/3158573190 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff178011d70 con 0x7ff1880b41f0 2023-10-06T01:41:24.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.076+0000 7ff17dffb700 1 -- 172.21.15.27:0/3158573190 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff178024440 con 0x7ff1880b41f0 2023-10-06T01:41:24.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.076+0000 7ff17dffb700 1 -- 172.21.15.27:0/3158573190 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1780247e0 con 0x7ff1880b41f0 2023-10-06T01:41:24.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.076+0000 7ff1900cf700 1 -- 172.21.15.27:0/3158573190 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff18806aba0 con 0x7ff1880b41f0 2023-10-06T01:41:24.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.077+0000 7ff1900cf700 1 -- 172.21.15.27:0/3158573190 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff18815c410 con 0x7ff1880b41f0 2023-10-06T01:41:24.123 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.077+0000 7f0d96bd5700 1 -- 172.21.15.27:0/1658799220 shutdown_connections 2023-10-06T01:41:24.123 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.077+0000 7f0d96bd5700 1 --2- 172.21.15.27:0/1658799220 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0d90159500 0x7f0d901598d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.123 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.077+0000 7f0d96bd5700 1 --2- 172.21.15.27:0/1658799220 >> v2:172.21.15.27:3300/0 conn(0x7f0d900b3fd0 0x7f0d9015d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.123 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.077+0000 7f0d96bd5700 1 -- 172.21.15.27:0/1658799220 >> 172.21.15.27:0/1658799220 conn(0x7f0d90067630 msgr2=0x7f0d90069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:24.123 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.078+0000 7f29939a9700 1 Processor -- start 2023-10-06T01:41:24.123 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.078+0000 7f29939a9700 1 -- start start 2023-10-06T01:41:24.124 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.079+0000 7fed6effd700 1 -- 172.21.15.27:0/1168158451 <== mon.2 v2:172.21.15.124:3301/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fed78021070 con 0x7fed88156550 2023-10-06T01:41:24.124 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.079+0000 7f0d96bd5700 1 -- 172.21.15.27:0/1658799220 shutdown_connections 2023-10-06T01:41:24.124 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.079+0000 7ff1737fe700 1 -- 172.21.15.27:0/3158573190 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1680052b0 con 0x7ff1880b41f0 2023-10-06T01:41:24.124 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.079+0000 7f0d96bd5700 1 -- 172.21.15.27:0/1658799220 wait complete. 2023-10-06T01:41:24.125 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.079+0000 7fed6effd700 1 --2- 172.21.15.27:0/1168158451 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fed74067000 0x7fed740694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.125 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.079+0000 7fed6effd700 1 -- 172.21.15.27:0/1168158451 <== mon.2 v2:172.21.15.124:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fed78085a80 con 0x7fed88156550 2023-10-06T01:41:24.125 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.079+0000 7f29939a9700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f298c0a69e0 0x7f298c16a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.125 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.079+0000 7f29939a9700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f298c0a6db0 0x7f298c0afaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.125 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.079+0000 7f29939a9700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f298c0aca20 0x7f298c069b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.126 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.079+0000 7fed86ffd700 1 --2- 172.21.15.27:0/1168158451 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fed74067000 0x7fed740694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.126 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.079+0000 7f29939a9700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298c0c76e0 con 0x7f298c0a69e0 2023-10-06T01:41:24.126 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.079+0000 7f29939a9700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298c16a870 con 0x7f298c0a6db0 2023-10-06T01:41:24.127 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.079+0000 7f29939a9700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f298c0af2c0 con 0x7f298c0aca20 2023-10-06T01:41:24.127 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.080+0000 7f2991745700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f298c0a69e0 0x7f298c16a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.127 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.080+0000 7f2991745700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f298c0a69e0 0x7f298c16a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56060/0 (socket says 172.21.15.27:56060) 2023-10-06T01:41:24.127 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.080+0000 7f2991745700 1 -- 172.21.15.27:0/2107892328 learned_addr learned my addr 172.21.15.27:0/2107892328 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:24.127 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.080+0000 7f2991f46700 1 --2- 172.21.15.27:0/2107892328 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f298c0aca20 0x7f298c069b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.127 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.080+0000 7fed86ffd700 1 --2- 172.21.15.27:0/1168158451 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fed74067000 0x7fed740694b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fed8000bbc0 tx=0x7fed8000b050 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:24.128 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.080+0000 7f2991745700 1 -- 172.21.15.27:0/2107892328 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f298c0aca20 msgr2=0x7f298c069b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.128 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.080+0000 7f2991745700 1 --2- 172.21.15.27:0/2107892328 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f298c0aca20 0x7f298c069b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.128 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.080+0000 7f2991745700 1 -- 172.21.15.27:0/2107892328 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f298c0a6db0 msgr2=0x7f298c0afaa0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-06T01:41:24.128 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.080+0000 7f2991745700 1 --2- 172.21.15.27:0/2107892328 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f298c0a6db0 0x7f298c0afaa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.128 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.080+0000 7f2991745700 1 -- 172.21.15.27:0/2107892328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2984009ae0 con 0x7f298c0a69e0 2023-10-06T01:41:24.129 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.080+0000 7f2991745700 1 --2- 172.21.15.27:0/2107892328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f298c0a69e0 0x7f298c16a330 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f297c00ec80 tx=0x7f297c009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:24.129 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.080+0000 7f29817fa700 1 -- 172.21.15.27:0/2107892328 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f297c00c850 con 0x7f298c0a69e0 2023-10-06T01:41:24.129 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.080+0000 7f29939a9700 1 -- 172.21.15.27:0/2107892328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f298c0c7930 con 0x7f298c0a69e0 2023-10-06T01:41:24.129 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.080+0000 7f29817fa700 1 -- 172.21.15.27:0/2107892328 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f297c00c9b0 con 0x7f298c0a69e0 2023-10-06T01:41:24.129 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.080+0000 7f29817fa700 1 -- 172.21.15.27:0/2107892328 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f297c00cd30 con 0x7f298c0a69e0 2023-10-06T01:41:24.129 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.081+0000 7f29939a9700 1 -- 172.21.15.27:0/2107892328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f298c0af580 con 0x7f298c0a69e0 2023-10-06T01:41:24.129 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.081+0000 7f0d96bd5700 1 Processor -- start 2023-10-06T01:41:24.129 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.081+0000 7f0d96bd5700 1 -- start start 2023-10-06T01:41:24.130 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.082+0000 7f29939a9700 1 -- 172.21.15.27:0/2107892328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f298c0c7e50 con 0x7f298c0a69e0 2023-10-06T01:41:24.130 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.082+0000 7fdb7effd700 1 --2- 172.21.15.27:0/45599546 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdb800cdde0 0x7fdb80142e90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdb7000e870 tx=0x7fdb70008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:24.130 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.082+0000 7fdb66ffd700 1 -- 172.21.15.27:0/45599546 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb70013410 con 0x7fdb800cdde0 2023-10-06T01:41:24.130 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.082+0000 7fdb66ffd700 1 -- 172.21.15.27:0/45599546 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb70013570 con 0x7fdb800cdde0 2023-10-06T01:41:24.130 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.082+0000 7fdb66ffd700 1 -- 172.21.15.27:0/45599546 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb70013930 con 0x7fdb800cdde0 2023-10-06T01:41:24.130 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.082+0000 7fdb8527a700 1 -- 172.21.15.27:0/45599546 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb8014a5c0 con 0x7fdb800cdde0 2023-10-06T01:41:24.130 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.082+0000 7fdb8527a700 1 -- 172.21.15.27:0/45599546 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb8014ab80 con 0x7fdb800cdde0 2023-10-06T01:41:24.131 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.082+0000 7f0d96bd5700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0d900b3fd0 0x7f0d9014cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.131 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.082+0000 7f0d96bd5700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0d90159500 0x7f0d90150e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.131 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.082+0000 7f0d96bd5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d9015d8e0 0x7f0d901513d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.132 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.082+0000 7f0d96bd5700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9016a9c0 con 0x7f0d9015d8e0 2023-10-06T01:41:24.132 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.082+0000 7f0d96bd5700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d90151a00 con 0x7f0d90159500 2023-10-06T01:41:24.132 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.082+0000 7f0d96bd5700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9015a8a0 con 0x7f0d900b3fd0 2023-10-06T01:41:24.132 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.082+0000 7f0d8ffff700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0d90159500 0x7f0d90150e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.132 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.083+0000 7f0d8ffff700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0d90159500 0x7f0d90150e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.27:43238/0 (socket says 172.21.15.27:43238) 2023-10-06T01:41:24.132 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.083+0000 7f0d8ffff700 1 -- 172.21.15.27:0/282373618 learned_addr learned my addr 172.21.15.27:0/282373618 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:24.132 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.083+0000 7f0d95172700 1 --2- 172.21.15.27:0/282373618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d9015d8e0 0x7f0d901513d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.132 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.083+0000 7f0d94971700 1 --2- 172.21.15.27:0/282373618 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0d900b3fd0 0x7f0d9014cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.132 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.083+0000 7f0d95172700 1 -- 172.21.15.27:0/282373618 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0d900b3fd0 msgr2=0x7f0d9014cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.132 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.083+0000 7f0d95172700 1 --2- 172.21.15.27:0/282373618 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0d900b3fd0 0x7f0d9014cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.132 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.083+0000 7f0d95172700 1 -- 172.21.15.27:0/282373618 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0d90159500 msgr2=0x7f0d90150e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.132 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.083+0000 7f0d95172700 1 --2- 172.21.15.27:0/282373618 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0d90159500 0x7f0d90150e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.133 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.083+0000 7f0d95172700 1 -- 172.21.15.27:0/282373618 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d80009420 con 0x7f0d9015d8e0 2023-10-06T01:41:24.133 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.083+0000 7f0d8ffff700 1 --2- 172.21.15.27:0/282373618 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0d90159500 0x7f0d90150e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:41:24.133 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.083+0000 7ff17dffb700 1 -- 172.21.15.27:0/3158573190 <== mon.2 v2:172.21.15.124:3301/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff17801a420 con 0x7ff1880b41f0 2023-10-06T01:41:24.133 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.083+0000 7f0d95172700 1 --2- 172.21.15.27:0/282373618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d9015d8e0 0x7f0d901513d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0d8800cb90 tx=0x7f0d88001980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:24.133 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.083+0000 7ff17dffb700 1 --2- 172.21.15.27:0/3158573190 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7ff16c066f30 0x7ff16c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.134 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.083+0000 7fdb8527a700 1 -- 172.21.15.27:0/45599546 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb80064050 con 0x7fdb800cdde0 2023-10-06T01:41:24.134 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.084+0000 7ff17dffb700 1 -- 172.21.15.27:0/3158573190 <== mon.2 v2:172.21.15.124:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7ff178086300 con 0x7ff1880b41f0 2023-10-06T01:41:24.135 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.084+0000 7f0d8cff9700 1 -- 172.21.15.27:0/282373618 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d8800a8b0 con 0x7f0d9015d8e0 2023-10-06T01:41:24.135 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.084+0000 7f0d8cff9700 1 -- 172.21.15.27:0/282373618 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d8800aa10 con 0x7f0d9015d8e0 2023-10-06T01:41:24.135 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.084+0000 7f0d8cff9700 1 -- 172.21.15.27:0/282373618 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d8800ad90 con 0x7f0d9015d8e0 2023-10-06T01:41:24.135 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.084+0000 7f0d96bd5700 1 -- 172.21.15.27:0/282373618 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d9015ab80 con 0x7f0d9015d8e0 2023-10-06T01:41:24.135 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.084+0000 7f0d96bd5700 1 -- 172.21.15.27:0/282373618 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d9015b140 con 0x7f0d9015d8e0 2023-10-06T01:41:24.135 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.084+0000 7f2b70511700 1 -- 172.21.15.27:0/345213413 shutdown_connections 2023-10-06T01:41:24.136 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.084+0000 7ff18d66a700 1 --2- 172.21.15.27:0/3158573190 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7ff16c066f30 0x7ff16c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.136 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.084+0000 7fdb66ffd700 1 -- 172.21.15.27:0/45599546 <== mon.2 v2:172.21.15.124:3301/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fdb70013ca0 con 0x7fdb800cdde0 2023-10-06T01:41:24.136 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.085+0000 7fed8d91b700 1 -- 172.21.15.27:0/1168158451 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed88172450 con 0x7fed88156550 2023-10-06T01:41:24.136 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.085+0000 7ff18d66a700 1 --2- 172.21.15.27:0/3158573190 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7ff16c066f30 0x7ff16c0693e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff18806ec10 tx=0x7ff184007ec0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:24.137 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.085+0000 7fdb66ffd700 1 --2- 172.21.15.27:0/45599546 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdb6c066f30 0x7fdb6c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.137 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.085+0000 7fdb66ffd700 1 -- 172.21.15.27:0/45599546 <== mon.2 v2:172.21.15.124:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fdb70084420 con 0x7fdb800cdde0 2023-10-06T01:41:24.137 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.085+0000 7fdb7e7fc700 1 --2- 172.21.15.27:0/45599546 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdb6c066f30 0x7fdb6c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.137 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.085+0000 7f29817fa700 1 -- 172.21.15.27:0/2107892328 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f297c013680 con 0x7f298c0a69e0 2023-10-06T01:41:24.138 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.086+0000 7fdb7e7fc700 1 --2- 172.21.15.27:0/45599546 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdb6c066f30 0x7fdb6c0693e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdb8006abe0 tx=0x7fdb74016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:24.138 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.086+0000 7f2b70511700 1 -- 172.21.15.27:0/345213413 wait complete. 2023-10-06T01:41:24.138 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.086+0000 7f0d8cff9700 1 -- 172.21.15.27:0/282373618 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0d88007300 con 0x7f0d9015d8e0 2023-10-06T01:41:24.138 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.086+0000 7f29817fa700 1 --2- 172.21.15.27:0/2107892328 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2970066f00 0x7f29700693b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.139 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.086+0000 7fba4c603700 1 Processor -- start 2023-10-06T01:41:24.139 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.086+0000 7fba4c603700 1 -- start start 2023-10-06T01:41:24.139 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.086+0000 7f29817fa700 1 -- 172.21.15.27:0/2107892328 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f297c084e00 con 0x7f298c0a69e0 2023-10-06T01:41:24.139 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.086+0000 7f0d8cff9700 1 --2- 172.21.15.27:0/282373618 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0d7c066fb0 0x7f0d7c069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.139 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.086+0000 7f0d8cff9700 1 -- 172.21.15.27:0/282373618 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f0d880842f0 con 0x7f0d9015d8e0 2023-10-06T01:41:24.140 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.086+0000 7f2990f44700 1 --2- 172.21.15.27:0/2107892328 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2970066f00 0x7f29700693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.140 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.087+0000 7f2b70511700 1 Processor -- start 2023-10-06T01:41:24.140 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.087+0000 7f2b70511700 1 -- start start 2023-10-06T01:41:24.141 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.088+0000 7fba4c603700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fba440a69e0 0x7fba440cb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.141 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.088+0000 7fba4c603700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fba440aca20 0x7fba440b1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.141 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.088+0000 7fba4c603700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba440add40 con 0x7fba440a6db0 2023-10-06T01:41:24.141 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.088+0000 7fba4c603700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba440ae8e0 con 0x7fba440a69e0 2023-10-06T01:41:24.141 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.088+0000 7fba4c603700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba4415b2e0 con 0x7fba440aca20 2023-10-06T01:41:24.141 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.088+0000 7fba4a39f700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fba440a69e0 0x7fba440cb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.142 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.088+0000 7fba4a39f700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fba440a69e0 0x7fba440cb6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56076/0 (socket says 172.21.15.27:56076) 2023-10-06T01:41:24.142 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.088+0000 7fba4a39f700 1 -- 172.21.15.27:0/737971840 learned_addr learned my addr 172.21.15.27:0/737971840 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:24.142 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.088+0000 7fba4aba0700 1 --2- 172.21.15.27:0/737971840 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fba440aca20 0x7fba440b1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.142 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.088+0000 7fba4a39f700 1 -- 172.21.15.27:0/737971840 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fba440aca20 msgr2=0x7fba440b1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:24.142 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.088+0000 7fba4a39f700 1 --2- 172.21.15.27:0/737971840 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fba440aca20 0x7fba440b1350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.142 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.088+0000 7fba4a39f700 1 -- 172.21.15.27:0/737971840 >> v1:172.21.15.27:6789/0 conn(0x7fba440a6db0 legacy=0x7fba440ae1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.142 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.088+0000 7fba4a39f700 1 -- 172.21.15.27:0/737971840 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba440ad410 con 0x7fba440a69e0 2023-10-06T01:41:24.142 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.088+0000 7f0d96bd5700 1 -- 172.21.15.27:0/282373618 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d9015b360 con 0x7f0d9015d8e0 2023-10-06T01:41:24.143 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.088+0000 7f2b70511700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b54003ae0 0x7f2b68165500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.143 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.088+0000 7f2b70511700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2b680af5a0 0x7f2b680c6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.143 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.088+0000 7f2b70511700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2b680c4430 0x7f2b680c36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.143 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.088+0000 7f2b70511700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b680af3b0 con 0x7f2b54003ae0 2023-10-06T01:41:24.143 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.088+0000 7f2b70511700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b680afa10 con 0x7f2b680c4430 2023-10-06T01:41:24.143 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.088+0000 7f2b70511700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b68165b50 con 0x7f2b680af5a0 2023-10-06T01:41:24.144 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.089+0000 7fba4a39f700 1 --2- 172.21.15.27:0/737971840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba440a69e0 0x7fba440cb6b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fba3c009f70 tx=0x7fba3c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61503e34b10884bd server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:24.144 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.089+0000 7f2b6e2ad700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b54003ae0 0x7f2b68165500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.145 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.089+0000 7f2b6daac700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2b680af5a0 0x7f2b680c6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.145 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.089+0000 7f2b6daac700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2b680af5a0 0x7f2b680c6760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.27:36098/0 (socket says 172.21.15.27:36098) 2023-10-06T01:41:24.145 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.089+0000 7f2b6e2ad700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b54003ae0 0x7f2b68165500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:56090/0 (socket says 172.21.15.27:56090) 2023-10-06T01:41:24.145 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.089+0000 7f2b6daac700 1 -- 172.21.15.27:0/2862687800 learned_addr learned my addr 172.21.15.27:0/2862687800 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:24.145 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.089+0000 7f2b6eaae700 1 --2- 172.21.15.27:0/2862687800 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2b680c4430 0x7f2b680c36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.145 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.089+0000 7f2b6daac700 1 -- 172.21.15.27:0/2862687800 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2b680c4430 msgr2=0x7f2b680c36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.145 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.089+0000 7f2b6daac700 1 --2- 172.21.15.27:0/2862687800 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2b680c4430 0x7f2b680c36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.145 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.089+0000 7f2b6daac700 1 -- 172.21.15.27:0/2862687800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b54003ae0 msgr2=0x7f2b68165500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.145 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.089+0000 7f2b6daac700 1 --2- 172.21.15.27:0/2862687800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b54003ae0 0x7f2b68165500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.145 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.089+0000 7f2b6daac700 1 -- 172.21.15.27:0/2862687800 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b68165e30 con 0x7f2b680af5a0 2023-10-06T01:41:24.145 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.089+0000 7f2b6e2ad700 1 --2- 172.21.15.27:0/2862687800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b54003ae0 0x7f2b68165500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:41:24.146 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.089+0000 7f2b6daac700 1 --2- 172.21.15.27:0/2862687800 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2b680af5a0 0x7f2b680c6760 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2b6000bc30 tx=0x7f2b60009820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:24.146 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.093+0000 7f0d94971700 1 --2- 172.21.15.27:0/282373618 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0d7c066fb0 0x7f0d7c069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.146 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.093+0000 7f2990f44700 1 --2- 172.21.15.27:0/2107892328 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2970066f00 0x7f29700693b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2988001050 tx=0x7f2988007ae0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:24.146 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.093+0000 7ff17dffb700 1 -- 172.21.15.27:0/3158573190 <== mon.2 v2:172.21.15.124:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff178053490 con 0x7ff1880b41f0 2023-10-06T01:41:24.147 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.093+0000 7f0d94971700 1 --2- 172.21.15.27:0/282373618 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0d7c066fb0 0x7f0d7c069460 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0d800093f0 tx=0x7f0d80017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:24.147 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.093+0000 7fdb66ffd700 1 -- 172.21.15.27:0/45599546 <== mon.2 v2:172.21.15.124:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdb70051530 con 0x7fdb800cdde0 2023-10-06T01:41:24.147 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.095+0000 7f29817fa700 1 -- 172.21.15.27:0/2107892328 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f297c051f10 con 0x7f298c0a69e0 2023-10-06T01:41:24.147 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.098+0000 7fed6effd700 1 -- 172.21.15.27:0/1168158451 <== mon.2 v2:172.21.15.124:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fed78028080 con 0x7fed88156550 2023-10-06T01:41:24.148 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.102+0000 7f0d8cff9700 1 -- 172.21.15.27:0/282373618 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0d8804b5a0 con 0x7f0d9015d8e0 2023-10-06T01:41:24.148 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.102+0000 7fba4939d700 1 -- 172.21.15.27:0/737971840 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba3c00e070 con 0x7fba440a69e0 2023-10-06T01:41:24.148 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.102+0000 7fba4939d700 1 -- 172.21.15.27:0/737971840 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba3c00bad0 con 0x7fba440a69e0 2023-10-06T01:41:24.148 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.102+0000 7fba4939d700 1 -- 172.21.15.27:0/737971840 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba3c0092e0 con 0x7fba440a69e0 2023-10-06T01:41:24.148 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.103+0000 7fba4c603700 1 -- 172.21.15.27:0/737971840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba440a69e0 msgr2=0x7fba440cb6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.148 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.103+0000 7fba4c603700 1 --2- 172.21.15.27:0/737971840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba440a69e0 0x7fba440cb6b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fba3c009f70 tx=0x7fba3c009840 comp rx=0 tx=0).stop 2023-10-06T01:41:24.148 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.104+0000 7fba4c603700 1 -- 172.21.15.27:0/737971840 shutdown_connections 2023-10-06T01:41:24.149 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.104+0000 7fba4c603700 1 --2- 172.21.15.27:0/737971840 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fba440aca20 0x7fba440b1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.149 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.104+0000 7fba4c603700 1 --2- 172.21.15.27:0/737971840 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba440a69e0 0x7fba440cb6b0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.149 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.104+0000 7fba4c603700 1 -- 172.21.15.27:0/737971840 >> 172.21.15.27:0/737971840 conn(0x7fba44068b90 msgr2=0x7fba440694b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:24.149 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.104+0000 7f2b5e7fc700 1 -- 172.21.15.27:0/2862687800 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b6000c840 con 0x7f2b680af5a0 2023-10-06T01:41:24.149 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.104+0000 7f2b5e7fc700 1 -- 172.21.15.27:0/2862687800 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b6000c9e0 con 0x7f2b680af5a0 2023-10-06T01:41:24.149 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.104+0000 7f2b5e7fc700 1 -- 172.21.15.27:0/2862687800 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b6000cda0 con 0x7f2b680af5a0 2023-10-06T01:41:24.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.104+0000 7fba4c603700 1 -- 172.21.15.27:0/737971840 shutdown_connections 2023-10-06T01:41:24.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.105+0000 7fba4c603700 1 -- 172.21.15.27:0/737971840 wait complete. 2023-10-06T01:41:24.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.106+0000 7fba4c603700 1 Processor -- start 2023-10-06T01:41:24.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.107+0000 7fba4c603700 1 -- start start 2023-10-06T01:41:24.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.108+0000 7fba4c603700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba440a69e0 0x7fba440a3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.108+0000 7fba4c603700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fba440a6db0 0x7fba44158960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.108+0000 7fba4c603700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba440aca20 0x7fba4406a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.108+0000 7fba4c603700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4416ee10 con 0x7fba440a69e0 2023-10-06T01:41:24.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.108+0000 7fba4c603700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4416e290 con 0x7fba440aca20 2023-10-06T01:41:24.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.108+0000 7fba4c603700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba4416e570 con 0x7fba440a6db0 2023-10-06T01:41:24.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.108+0000 7fba49b9e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fba440a6db0 0x7fba44158960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.108+0000 7fba49b9e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fba440a6db0 0x7fba44158960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.27:36106/0 (socket says 172.21.15.27:36106) 2023-10-06T01:41:24.152 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.108+0000 7fba49b9e700 1 -- 172.21.15.27:0/2386267352 learned_addr learned my addr 172.21.15.27:0/2386267352 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:24.152 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.109+0000 7fba4a39f700 1 --2- 172.21.15.27:0/2386267352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba440a69e0 0x7fba440a3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.152 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.109+0000 7fba4a39f700 1 -- 172.21.15.27:0/2386267352 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fba440a6db0 msgr2=0x7fba44158960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.152 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.109+0000 7fba4a39f700 1 --2- 172.21.15.27:0/2386267352 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fba440a6db0 0x7fba44158960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.152 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.109+0000 7fba4a39f700 1 -- 172.21.15.27:0/2386267352 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba440aca20 msgr2=0x7fba4406a600 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-06T01:41:24.152 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.109+0000 7fba4a39f700 1 --2- 172.21.15.27:0/2386267352 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba440aca20 0x7fba4406a600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.152 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.109+0000 7fba4a39f700 1 -- 172.21.15.27:0/2386267352 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba3c009ae0 con 0x7fba440a69e0 2023-10-06T01:41:24.152 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.109+0000 7fba49b9e700 1 --2- 172.21.15.27:0/2386267352 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fba440a6db0 0x7fba44158960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:41:24.152 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.109+0000 7fba4a39f700 1 --2- 172.21.15.27:0/2386267352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba440a69e0 0x7fba440a3a80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fba3c015000 tx=0x7fba3c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:24.152 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.110+0000 7fba3a7fc700 1 -- 172.21.15.27:0/2386267352 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba3c008a80 con 0x7fba440a69e0 2023-10-06T01:41:24.152 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.110+0000 7fba3a7fc700 1 -- 172.21.15.27:0/2386267352 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba3c008be0 con 0x7fba440a69e0 2023-10-06T01:41:24.153 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.110+0000 7fba3a7fc700 1 -- 172.21.15.27:0/2386267352 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba3c026460 con 0x7fba440a69e0 2023-10-06T01:41:24.153 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.110+0000 7fba4c603700 1 -- 172.21.15.27:0/2386267352 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba4416e850 con 0x7fba440a69e0 2023-10-06T01:41:24.153 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.110+0000 7fba4c603700 1 -- 172.21.15.27:0/2386267352 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba440a3600 con 0x7fba440a69e0 2023-10-06T01:41:24.153 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.113+0000 7fba3a7fc700 1 -- 172.21.15.27:0/2386267352 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fba3c003040 con 0x7fba440a69e0 2023-10-06T01:41:24.153 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.113+0000 7fba3a7fc700 1 --2- 172.21.15.27:0/2386267352 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fba28067080 0x7fba28069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.153 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.113+0000 7fba3a7fc700 1 -- 172.21.15.27:0/2386267352 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fba3c0864b0 con 0x7fba440a69e0 2023-10-06T01:41:24.153 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.108+0000 7f2b70511700 1 -- 172.21.15.27:0/2862687800 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b68166110 con 0x7f2b680af5a0 2023-10-06T01:41:24.153 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.108+0000 7f2b70511700 1 -- 172.21.15.27:0/2862687800 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b680adc30 con 0x7f2b680af5a0 2023-10-06T01:41:24.154 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.114+0000 7f2b70511700 1 -- 172.21.15.27:0/2862687800 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b68063420 con 0x7f2b680af5a0 2023-10-06T01:41:24.154 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.116+0000 7fba4c603700 1 -- 172.21.15.27:0/2386267352 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba44064050 con 0x7fba440a69e0 2023-10-06T01:41:24.154 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.117+0000 7fba49b9e700 1 --2- 172.21.15.27:0/2386267352 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fba28067080 0x7fba28069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.154 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.117+0000 7fba49b9e700 1 --2- 172.21.15.27:0/2386267352 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fba28067080 0x7fba28069530 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fba440c8780 tx=0x7fba34009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:24.154 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.121+0000 7f2b5e7fc700 1 -- 172.21.15.27:0/2862687800 <== mon.2 v2:172.21.15.124:3301/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2b60014620 con 0x7f2b680af5a0 2023-10-06T01:41:24.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.121+0000 7f2b5e7fc700 1 --2- 172.21.15.27:0/2862687800 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2b4c067000 0x7f2b4c0694b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:24.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.121+0000 7f2b5e7fc700 1 -- 172.21.15.27:0/2862687800 <== mon.2 v2:172.21.15.124:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f2b600846f0 con 0x7f2b680af5a0 2023-10-06T01:41:24.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.124+0000 7f2b6e2ad700 1 --2- 172.21.15.27:0/2862687800 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2b4c067000 0x7f2b4c0694b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:24.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.124+0000 7f2b6e2ad700 1 --2- 172.21.15.27:0/2862687800 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2b4c067000 0x7f2b4c0694b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2b64000c00 tx=0x7f2b64015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:24.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.127+0000 7fba3a7fc700 1 -- 172.21.15.27:0/2386267352 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fba3c0535a0 con 0x7fba440a69e0 2023-10-06T01:41:24.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.134+0000 7f2b5e7fc700 1 -- 172.21.15.27:0/2862687800 <== mon.2 v2:172.21.15.124:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2b6004c5a0 con 0x7f2b680af5a0 2023-10-06T01:41:24.466 INFO:teuthology.orchestra.run.smithi027.stdout:55834574855 2023-10-06T01:41:24.466 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.453+0000 7f882e95b700 1 -- 172.21.15.27:0/1403943215 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f8808005830 con 0x7f88280ab3b0 2023-10-06T01:41:24.466 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.463+0000 7f880ffff700 1 -- 172.21.15.27:0/1403943215 <== mon.1 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8820012320 con 0x7f88280ab3b0 2023-10-06T01:41:24.473 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.471+0000 7f882e95b700 1 -- 172.21.15.27:0/1403943215 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8814066f30 msgr2=0x7f88140693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.473 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.471+0000 7f882e95b700 1 --2- 172.21.15.27:0/1403943215 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8814066f30 0x7f88140693e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f882806f800 tx=0x7f8818008040 comp rx=0 tx=0).stop 2023-10-06T01:41:24.474 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.471+0000 7f882e95b700 1 -- 172.21.15.27:0/1403943215 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f88280ab3b0 msgr2=0x7f8828169cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.474 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.471+0000 7f882e95b700 1 --2- 172.21.15.27:0/1403943215 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f88280ab3b0 0x7f8828169cf0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f882000bc90 tx=0x7f8820009840 comp rx=0 tx=0).stop 2023-10-06T01:41:24.474 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.472+0000 7f882e95b700 1 -- 172.21.15.27:0/1403943215 shutdown_connections 2023-10-06T01:41:24.474 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.472+0000 7f882e95b700 1 --2- 172.21.15.27:0/1403943215 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8814066f30 0x7f88140693e0 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f882806f800 tx=0x7f8818008040 comp rx=0 tx=0).stop 2023-10-06T01:41:24.474 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.472+0000 7f882e95b700 1 --2- 172.21.15.27:0/1403943215 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8828156930 0x7f88280b15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.474 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.472+0000 7f882e95b700 1 --2- 172.21.15.27:0/1403943215 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f88280ab3b0 0x7f8828169cf0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.474 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.472+0000 7f882e95b700 1 --2- 172.21.15.27:0/1403943215 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f88280cb980 0x7f88280a73a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.475 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.472+0000 7f882e95b700 1 -- 172.21.15.27:0/1403943215 >> 172.21.15.27:0/1403943215 conn(0x7f8828068b90 msgr2=0x7f88281556e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:24.476 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.476+0000 7f882e95b700 1 -- 172.21.15.27:0/1403943215 shutdown_connections 2023-10-06T01:41:24.476 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.476+0000 7f882e95b700 1 -- 172.21.15.27:0/1403943215 wait complete. 2023-10-06T01:41:24.494 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574855 for osd.2 2023-10-06T01:41:24.495 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.494+0000 7fdb8527a700 1 -- 172.21.15.27:0/45599546 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fdb800aeae0 con 0x7fdb800cdde0 2023-10-06T01:41:24.495 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.494+0000 7fdb66ffd700 1 -- 172.21.15.27:0/45599546 <== mon.2 v2:172.21.15.124:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdb70054b50 con 0x7fdb800cdde0 2023-10-06T01:41:24.495 INFO:teuthology.orchestra.run.smithi027.stdout:55834574857 2023-10-06T01:41:24.497 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.497+0000 7fed8d91b700 1 -- 172.21.15.27:0/1168158451 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fed88064050 con 0x7fed88156550 2023-10-06T01:41:24.498 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.498+0000 7fdb8527a700 1 -- 172.21.15.27:0/45599546 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdb6c066f30 msgr2=0x7fdb6c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.498 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.498+0000 7fdb8527a700 1 --2- 172.21.15.27:0/45599546 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdb6c066f30 0x7fdb6c0693e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdb8006abe0 tx=0x7fdb74016000 comp rx=0 tx=0).stop 2023-10-06T01:41:24.498 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.498+0000 7fdb8527a700 1 -- 172.21.15.27:0/45599546 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdb800cdde0 msgr2=0x7fdb80142e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.498 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.498+0000 7fdb8527a700 1 --2- 172.21.15.27:0/45599546 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdb800cdde0 0x7fdb80142e90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdb7000e870 tx=0x7fdb70008040 comp rx=0 tx=0).stop 2023-10-06T01:41:24.499 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.498+0000 7fed6effd700 1 -- 172.21.15.27:0/1168158451 <== mon.2 v2:172.21.15.124:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fed78052cc0 con 0x7fed88156550 2023-10-06T01:41:24.499 INFO:teuthology.orchestra.run.smithi027.stdout:55834574854 2023-10-06T01:41:24.500 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.500+0000 7fed8d91b700 1 -- 172.21.15.27:0/1168158451 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fed74067000 msgr2=0x7fed740694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.500 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.500+0000 7fed8d91b700 1 --2- 172.21.15.27:0/1168158451 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fed74067000 0x7fed740694b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fed8000bbc0 tx=0x7fed8000b050 comp rx=0 tx=0).stop 2023-10-06T01:41:24.500 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.500+0000 7fed8d91b700 1 -- 172.21.15.27:0/1168158451 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fed88156550 msgr2=0x7fed88152440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.500+0000 7fed8d91b700 1 --2- 172.21.15.27:0/1168158451 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fed88156550 0x7fed88152440 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fed78009290 tx=0x7fed78016000 comp rx=0 tx=0).stop 2023-10-06T01:41:24.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.502+0000 7fdb8527a700 1 -- 172.21.15.27:0/45599546 shutdown_connections 2023-10-06T01:41:24.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.502+0000 7fdb8527a700 1 --2- 172.21.15.27:0/45599546 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdb6c066f30 0x7fdb6c0693e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.502+0000 7fdb8527a700 1 --2- 172.21.15.27:0/45599546 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdb800cdde0 0x7fdb80142e90 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.502+0000 7fdb8527a700 1 --2- 172.21.15.27:0/45599546 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdb801433d0 0x7fdb8014efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.502+0000 7fdb8527a700 1 --2- 172.21.15.27:0/45599546 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb8015d920 0x7fdb8014ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.503 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.502+0000 7fdb8527a700 1 -- 172.21.15.27:0/45599546 >> 172.21.15.27:0/45599546 conn(0x7fdb8006d890 msgr2=0x7fdb800b1220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:24.504 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.502+0000 7ff1737fe700 1 -- 172.21.15.27:0/3158573190 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7ff168005830 con 0x7ff1880b41f0 2023-10-06T01:41:24.504 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.502+0000 7ff17dffb700 1 -- 172.21.15.27:0/3158573190 <== mon.2 v2:172.21.15.124:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff178009350 con 0x7ff1880b41f0 2023-10-06T01:41:24.504 INFO:teuthology.orchestra.run.smithi027.stdout:55834574856 2023-10-06T01:41:24.505 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.503+0000 7f29939a9700 1 -- 172.21.15.27:0/2107892328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f298c0af7a0 con 0x7f298c0a69e0 2023-10-06T01:41:24.505 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.503+0000 7f29817fa700 1 -- 172.21.15.27:0/2107892328 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f297c017090 con 0x7f298c0a69e0 2023-10-06T01:41:24.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.504+0000 7f0d96bd5700 1 -- 172.21.15.27:0/282373618 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f0d9014d6e0 con 0x7f0d9015d8e0 2023-10-06T01:41:24.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.504+0000 7fed8d91b700 1 -- 172.21.15.27:0/1168158451 shutdown_connections 2023-10-06T01:41:24.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.504+0000 7fed8d91b700 1 --2- 172.21.15.27:0/1168158451 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fed74067000 0x7fed740694b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.504+0000 7fed8d91b700 1 --2- 172.21.15.27:0/1168158451 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fed88156550 0x7fed88152440 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.504+0000 7fed8d91b700 1 --2- 172.21.15.27:0/1168158451 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fed88069120 0x7fed880cb7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.504+0000 7fed8d91b700 1 --2- 172.21.15.27:0/1168158451 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fed880694f0 0x7fed8806c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.504+0000 7fed8d91b700 1 -- 172.21.15.27:0/1168158451 >> 172.21.15.27:0/1168158451 conn(0x7fed8806e910 msgr2=0x7fed880686f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:24.508 INFO:teuthology.orchestra.run.smithi027.stdout:55834574855 2023-10-06T01:41:24.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.504+0000 7f0d8cff9700 1 -- 172.21.15.27:0/282373618 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0d88017070 con 0x7f0d9015d8e0 2023-10-06T01:41:24.509 INFO:teuthology.orchestra.run.smithi027.stdout:55834574855 2023-10-06T01:41:24.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.506+0000 7ff1737fe700 1 -- 172.21.15.27:0/3158573190 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7ff16c066f30 msgr2=0x7ff16c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.506+0000 7ff1737fe700 1 --2- 172.21.15.27:0/3158573190 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7ff16c066f30 0x7ff16c0693e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff18806ec10 tx=0x7ff184007ec0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.506+0000 7ff1737fe700 1 -- 172.21.15.27:0/3158573190 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff1880b41f0 msgr2=0x7ff18815bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.506+0000 7ff1737fe700 1 --2- 172.21.15.27:0/3158573190 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff1880b41f0 0x7ff18815bed0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff1880afce0 tx=0x7ff178017000 comp rx=0 tx=0).stop 2023-10-06T01:41:24.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.506+0000 7fed8d91b700 1 -- 172.21.15.27:0/1168158451 shutdown_connections 2023-10-06T01:41:24.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.506+0000 7ff1737fe700 1 -- 172.21.15.27:0/3158573190 shutdown_connections 2023-10-06T01:41:24.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.506+0000 7ff1737fe700 1 --2- 172.21.15.27:0/3158573190 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7ff16c066f30 0x7ff16c0693e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.506+0000 7ff1737fe700 1 --2- 172.21.15.27:0/3158573190 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff1880b41f0 0x7ff18815bed0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.506+0000 7ff1737fe700 1 --2- 172.21.15.27:0/3158573190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff18815d890 0x7ff18806cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.506+0000 7ff1737fe700 1 --2- 172.21.15.27:0/3158573190 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff1880b4b90 0x7ff1880af2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.506+0000 7ff1737fe700 1 -- 172.21.15.27:0/3158573190 >> 172.21.15.27:0/3158573190 conn(0x7ff1880680b0 msgr2=0x7ff1880aff40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:24.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.506+0000 7fed8d91b700 1 -- 172.21.15.27:0/1168158451 wait complete. 2023-10-06T01:41:24.513 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.506+0000 7f29939a9700 1 -- 172.21.15.27:0/2107892328 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2970066f00 msgr2=0x7f29700693b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.513 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.506+0000 7f29939a9700 1 --2- 172.21.15.27:0/2107892328 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2970066f00 0x7f29700693b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2988001050 tx=0x7f2988007ae0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.513 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.506+0000 7f29939a9700 1 -- 172.21.15.27:0/2107892328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f298c0a69e0 msgr2=0x7f298c16a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.513 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.506+0000 7f29939a9700 1 --2- 172.21.15.27:0/2107892328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f298c0a69e0 0x7f298c16a330 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f297c00ec80 tx=0x7f297c009120 comp rx=0 tx=0).stop 2023-10-06T01:41:24.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.507+0000 7ff1737fe700 1 -- 172.21.15.27:0/3158573190 shutdown_connections 2023-10-06T01:41:24.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.507+0000 7ff1737fe700 1 -- 172.21.15.27:0/3158573190 wait complete. 2023-10-06T01:41:24.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.507+0000 7f29939a9700 1 -- 172.21.15.27:0/2107892328 shutdown_connections 2023-10-06T01:41:24.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.507+0000 7f29939a9700 1 --2- 172.21.15.27:0/2107892328 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2970066f00 0x7f29700693b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.507+0000 7f29939a9700 1 --2- 172.21.15.27:0/2107892328 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f298c0aca20 0x7f298c069b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.507+0000 7f29939a9700 1 --2- 172.21.15.27:0/2107892328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f298c0a69e0 0x7f298c16a330 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.507+0000 7f29939a9700 1 --2- 172.21.15.27:0/2107892328 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f298c0a6db0 0x7f298c0afaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.507+0000 7f29939a9700 1 -- 172.21.15.27:0/2107892328 >> 172.21.15.27:0/2107892328 conn(0x7f298c068b90 msgr2=0x7f298c15a9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:24.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.507+0000 7f29939a9700 1 -- 172.21.15.27:0/2107892328 shutdown_connections 2023-10-06T01:41:24.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.507+0000 7f29939a9700 1 -- 172.21.15.27:0/2107892328 wait complete. 2023-10-06T01:41:24.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.507+0000 7fdb8527a700 1 -- 172.21.15.27:0/45599546 shutdown_connections 2023-10-06T01:41:24.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.507+0000 7f0d96bd5700 1 -- 172.21.15.27:0/282373618 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0d7c066fb0 msgr2=0x7f0d7c069460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.507+0000 7f0d96bd5700 1 --2- 172.21.15.27:0/282373618 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0d7c066fb0 0x7f0d7c069460 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0d800093f0 tx=0x7f0d80017000 comp rx=0 tx=0).stop 2023-10-06T01:41:24.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.507+0000 7f0d96bd5700 1 -- 172.21.15.27:0/282373618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d9015d8e0 msgr2=0x7f0d901513d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.507+0000 7f0d96bd5700 1 --2- 172.21.15.27:0/282373618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d9015d8e0 0x7f0d901513d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0d8800cb90 tx=0x7f0d88001980 comp rx=0 tx=0).stop 2023-10-06T01:41:24.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.507+0000 7f0d96bd5700 1 -- 172.21.15.27:0/282373618 shutdown_connections 2023-10-06T01:41:24.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.507+0000 7f0d96bd5700 1 --2- 172.21.15.27:0/282373618 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0d7c066fb0 0x7f0d7c069460 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.507+0000 7f0d96bd5700 1 --2- 172.21.15.27:0/282373618 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0d900b3fd0 0x7f0d9014cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.507+0000 7f0d96bd5700 1 --2- 172.21.15.27:0/282373618 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d9015d8e0 0x7f0d901513d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.507+0000 7f0d96bd5700 1 --2- 172.21.15.27:0/282373618 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0d90159500 0x7f0d90150e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.507+0000 7f0d96bd5700 1 -- 172.21.15.27:0/282373618 >> 172.21.15.27:0/282373618 conn(0x7f0d90067630 msgr2=0x7f0d900b4440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:24.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.508+0000 7fdb8527a700 1 -- 172.21.15.27:0/45599546 wait complete. 2023-10-06T01:41:24.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.508+0000 7f0d96bd5700 1 -- 172.21.15.27:0/282373618 shutdown_connections 2023-10-06T01:41:24.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.508+0000 7f0d96bd5700 1 -- 172.21.15.27:0/282373618 wait complete. 2023-10-06T01:41:24.520 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.520+0000 7f2b70511700 1 -- 172.21.15.27:0/2862687800 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f2b680ade50 con 0x7f2b680af5a0 2023-10-06T01:41:24.520 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.520+0000 7f2b5e7fc700 1 -- 172.21.15.27:0/2862687800 <== mon.2 v2:172.21.15.124:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2b600517e0 con 0x7f2b680af5a0 2023-10-06T01:41:24.521 INFO:teuthology.orchestra.run.smithi027.stdout:55834574857 2023-10-06T01:41:24.525 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.1 2023-10-06T01:41:24.525 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.525+0000 7f2b70511700 1 -- 172.21.15.27:0/2862687800 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2b4c067000 msgr2=0x7f2b4c0694b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.525+0000 7f2b70511700 1 --2- 172.21.15.27:0/2862687800 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2b4c067000 0x7f2b4c0694b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2b64000c00 tx=0x7f2b64015000 comp rx=0 tx=0).stop 2023-10-06T01:41:24.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.525+0000 7f2b70511700 1 -- 172.21.15.27:0/2862687800 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2b680af5a0 msgr2=0x7f2b680c6760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.525+0000 7f2b70511700 1 --2- 172.21.15.27:0/2862687800 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2b680af5a0 0x7f2b680c6760 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2b6000bc30 tx=0x7f2b60009820 comp rx=0 tx=0).stop 2023-10-06T01:41:24.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.525+0000 7f2b70511700 1 -- 172.21.15.27:0/2862687800 shutdown_connections 2023-10-06T01:41:24.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.525+0000 7f2b70511700 1 --2- 172.21.15.27:0/2862687800 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2b4c067000 0x7f2b4c0694b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.525+0000 7f2b70511700 1 --2- 172.21.15.27:0/2862687800 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2b680c4430 0x7f2b680c36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.525+0000 7f2b70511700 1 --2- 172.21.15.27:0/2862687800 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2b54003ae0 0x7f2b68165500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.525+0000 7f2b70511700 1 --2- 172.21.15.27:0/2862687800 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2b680af5a0 0x7f2b680c6760 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.525+0000 7f2b70511700 1 -- 172.21.15.27:0/2862687800 >> 172.21.15.27:0/2862687800 conn(0x7f2b6806e760 msgr2=0x7f2b680c2880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:24.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.525+0000 7f2b70511700 1 -- 172.21.15.27:0/2862687800 shutdown_connections 2023-10-06T01:41:24.527 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.525+0000 7f2b70511700 1 -- 172.21.15.27:0/2862687800 wait complete. 2023-10-06T01:41:24.527 DEBUG:teuthology.parallel:result is None 2023-10-06T01:41:24.531 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.5 2023-10-06T01:41:24.531 DEBUG:teuthology.parallel:result is None 2023-10-06T01:41:24.532 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.4 2023-10-06T01:41:24.532 DEBUG:teuthology.parallel:result is None 2023-10-06T01:41:24.533 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.0 2023-10-06T01:41:24.534 DEBUG:teuthology.parallel:result is None 2023-10-06T01:41:24.536 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.536+0000 7fba4c603700 1 -- 172.21.15.27:0/2386267352 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fba440ad6f0 con 0x7fba440a69e0 2023-10-06T01:41:24.536 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.536+0000 7fba3a7fc700 1 -- 172.21.15.27:0/2386267352 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fba3c021070 con 0x7fba440a69e0 2023-10-06T01:41:24.536 INFO:teuthology.orchestra.run.smithi027.stdout:55834574855 2023-10-06T01:41:24.537 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.7 2023-10-06T01:41:24.537 DEBUG:teuthology.parallel:result is None 2023-10-06T01:41:24.539 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.538+0000 7fba4c603700 1 -- 172.21.15.27:0/2386267352 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fba28067080 msgr2=0x7fba28069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.539 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.538+0000 7fba4c603700 1 --2- 172.21.15.27:0/2386267352 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fba28067080 0x7fba28069530 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fba440c8780 tx=0x7fba34009040 comp rx=0 tx=0).stop 2023-10-06T01:41:24.539 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.538+0000 7fba4c603700 1 -- 172.21.15.27:0/2386267352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba440a69e0 msgr2=0x7fba440a3a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:24.539 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.538+0000 7fba4c603700 1 --2- 172.21.15.27:0/2386267352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba440a69e0 0x7fba440a3a80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fba3c015000 tx=0x7fba3c018000 comp rx=0 tx=0).stop 2023-10-06T01:41:24.539 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.538+0000 7fba4c603700 1 -- 172.21.15.27:0/2386267352 shutdown_connections 2023-10-06T01:41:24.539 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.538+0000 7fba4c603700 1 --2- 172.21.15.27:0/2386267352 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fba28067080 0x7fba28069530 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.539 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.538+0000 7fba4c603700 1 --2- 172.21.15.27:0/2386267352 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fba440aca20 0x7fba4406a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.539 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.538+0000 7fba4c603700 1 --2- 172.21.15.27:0/2386267352 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fba440a69e0 0x7fba440a3a80 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.540 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.538+0000 7fba4c603700 1 --2- 172.21.15.27:0/2386267352 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fba440a6db0 0x7fba44158960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:24.540 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.538+0000 7fba4c603700 1 -- 172.21.15.27:0/2386267352 >> 172.21.15.27:0/2386267352 conn(0x7fba44068b90 msgr2=0x7fba4415a830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:24.540 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.538+0000 7fba4c603700 1 -- 172.21.15.27:0/2386267352 shutdown_connections 2023-10-06T01:41:24.540 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:24.538+0000 7fba4c603700 1 -- 172.21.15.27:0/2386267352 wait complete. 2023-10-06T01:41:24.541 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.6 2023-10-06T01:41:24.541 DEBUG:teuthology.parallel:result is None 2023-10-06T01:41:24.551 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.3 2023-10-06T01:41:24.551 DEBUG:teuthology.parallel:result is None 2023-10-06T01:41:25.495 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-06T01:41:25.627 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.625+0000 7f37acab5700 1 Processor -- start 2023-10-06T01:41:25.627 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.625+0000 7f37acab5700 1 -- start start 2023-10-06T01:41:25.636 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.626+0000 7f37acab5700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f37a8159500 0x7f37a81598d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:25.637 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.626+0000 7f37acab5700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f37a80b3fd0 0x7f37a815d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:25.637 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.626+0000 7f37acab5700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37a815dd20 con 0x7f37a815d8e0 2023-10-06T01:41:25.637 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.626+0000 7f37acab5700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37a80b1110 con 0x7f37a8159500 2023-10-06T01:41:25.637 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.626+0000 7f37acab5700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37a80b13b0 con 0x7f37a80b3fd0 2023-10-06T01:41:25.637 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.626+0000 7f37a659c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f37a8159500 0x7f37a81598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:25.637 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.626+0000 7f37a6d9d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f37a815d8e0 0x7f37a80cf6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57016/0 (socket says 172.21.15.27:57016) 2023-10-06T01:41:25.637 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.626+0000 7f37a6d9d700 1 -- 172.21.15.27:0/1077490387 learned_addr learned my addr 172.21.15.27:0/1077490387 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:25.638 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.627+0000 7f37a5d9b700 1 --2- 172.21.15.27:0/1077490387 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f37a80b3fd0 0x7f37a815d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:25.638 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.627+0000 7f37a659c700 1 -- 172.21.15.27:0/1077490387 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f37a80b3fd0 msgr2=0x7f37a815d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:25.638 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.627+0000 7f37a659c700 1 --2- 172.21.15.27:0/1077490387 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f37a80b3fd0 0x7f37a815d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:25.638 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.627+0000 7f37a659c700 1 -- 172.21.15.27:0/1077490387 >> v1:172.21.15.27:6789/0 conn(0x7f37a815d8e0 legacy=0x7f37a80cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:25.638 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.627+0000 7f37a659c700 1 -- 172.21.15.27:0/1077490387 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a80b1690 con 0x7f37a8159500 2023-10-06T01:41:25.638 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.627+0000 7f37a559a700 1 -- 172.21.15.27:0/1077490387 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2423726357 0 0) 0x7f37a815dd20 con 0x7f37a815d8e0 2023-10-06T01:41:25.638 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.627+0000 7f37a5d9b700 1 --2- 172.21.15.27:0/1077490387 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f37a80b3fd0 0x7f37a815d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:41:25.639 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.627+0000 7f37a659c700 1 --2- 172.21.15.27:0/1077490387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37a8159500 0x7f37a81598d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f37a00098e0 tx=0x7f37a00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc1da3ac4b5607a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:25.639 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.627+0000 7f37a559a700 1 -- 172.21.15.27:0/1077490387 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a0009e80 con 0x7f37a8159500 2023-10-06T01:41:25.639 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.627+0000 7f37a559a700 1 -- 172.21.15.27:0/1077490387 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37a000b860 con 0x7f37a8159500 2023-10-06T01:41:25.639 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.627+0000 7f37a559a700 1 -- 172.21.15.27:0/1077490387 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a000bbe0 con 0x7f37a8159500 2023-10-06T01:41:25.639 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.628+0000 7f37acab5700 1 -- 172.21.15.27:0/1077490387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37a8159500 msgr2=0x7f37a81598d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:25.639 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.628+0000 7f37acab5700 1 --2- 172.21.15.27:0/1077490387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37a8159500 0x7f37a81598d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f37a00098e0 tx=0x7f37a00091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:25.639 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.628+0000 7f37acab5700 1 -- 172.21.15.27:0/1077490387 shutdown_connections 2023-10-06T01:41:25.639 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.628+0000 7f37acab5700 1 --2- 172.21.15.27:0/1077490387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37a8159500 0x7f37a81598d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:25.640 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.628+0000 7f37acab5700 1 --2- 172.21.15.27:0/1077490387 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f37a80b3fd0 0x7f37a815d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:25.640 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.628+0000 7f37acab5700 1 -- 172.21.15.27:0/1077490387 >> 172.21.15.27:0/1077490387 conn(0x7f37a8067630 msgr2=0x7f37a8069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:25.640 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.628+0000 7f37acab5700 1 -- 172.21.15.27:0/1077490387 shutdown_connections 2023-10-06T01:41:25.640 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.628+0000 7f37acab5700 1 -- 172.21.15.27:0/1077490387 wait complete. 2023-10-06T01:41:25.640 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.629+0000 7f37acab5700 1 Processor -- start 2023-10-06T01:41:25.640 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.629+0000 7f37acab5700 1 -- start start 2023-10-06T01:41:25.640 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.630+0000 7f37acab5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37a80b3fd0 0x7f37a8140050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:25.640 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.630+0000 7f37acab5700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f37a8159500 0x7f37a816d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:25.640 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.630+0000 7f37acab5700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f37a815d8e0 0x7f37a816db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:25.640 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.630+0000 7f37acab5700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a8159d10 con 0x7f37a80b3fd0 2023-10-06T01:41:25.640 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.630+0000 7f37acab5700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a8169fa0 con 0x7f37a8159500 2023-10-06T01:41:25.640 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.630+0000 7f37acab5700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37a815a570 con 0x7f37a815d8e0 2023-10-06T01:41:25.640 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.630+0000 7f37a659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37a80b3fd0 0x7f37a8140050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:25.641 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.630+0000 7f37a659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37a80b3fd0 0x7f37a8140050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52236/0 (socket says 172.21.15.27:52236) 2023-10-06T01:41:25.641 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.630+0000 7f37a659c700 1 -- 172.21.15.27:0/3167823609 learned_addr learned my addr 172.21.15.27:0/3167823609 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:25.641 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.630+0000 7f37a6d9d700 1 --2- 172.21.15.27:0/3167823609 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f37a815d8e0 0x7f37a816db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:25.641 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.630+0000 7f37a5d9b700 1 --2- 172.21.15.27:0/3167823609 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f37a8159500 0x7f37a816d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:25.641 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.630+0000 7f37a659c700 1 -- 172.21.15.27:0/3167823609 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f37a815d8e0 msgr2=0x7f37a816db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:25.641 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.630+0000 7f37a659c700 1 --2- 172.21.15.27:0/3167823609 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f37a815d8e0 0x7f37a816db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:25.641 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.630+0000 7f37a659c700 1 -- 172.21.15.27:0/3167823609 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f37a8159500 msgr2=0x7f37a816d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:25.641 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.630+0000 7f37a659c700 1 --2- 172.21.15.27:0/3167823609 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f37a8159500 0x7f37a816d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:25.641 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.630+0000 7f37a659c700 1 -- 172.21.15.27:0/3167823609 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a0009420 con 0x7f37a80b3fd0 2023-10-06T01:41:25.641 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.630+0000 7f37a6d9d700 1 --2- 172.21.15.27:0/3167823609 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f37a815d8e0 0x7f37a816db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:41:25.641 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.630+0000 7f37a5d9b700 1 --2- 172.21.15.27:0/3167823609 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f37a8159500 0x7f37a816d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:41:25.641 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.630+0000 7f37a659c700 1 --2- 172.21.15.27:0/3167823609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37a80b3fd0 0x7f37a8140050 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f37a815deb0 tx=0x7f37a0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:25.642 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.630+0000 7f379e7fc700 1 -- 172.21.15.27:0/3167823609 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a000bbe0 con 0x7f37a80b3fd0 2023-10-06T01:41:25.642 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.630+0000 7f379e7fc700 1 -- 172.21.15.27:0/3167823609 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37a0011420 con 0x7f37a80b3fd0 2023-10-06T01:41:25.642 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.630+0000 7f37acab5700 1 -- 172.21.15.27:0/3167823609 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37a815ffd0 con 0x7f37a80b3fd0 2023-10-06T01:41:25.642 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.630+0000 7f379e7fc700 1 -- 172.21.15.27:0/3167823609 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a00117c0 con 0x7f37a80b3fd0 2023-10-06T01:41:25.642 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.630+0000 7f37acab5700 1 -- 172.21.15.27:0/3167823609 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37a815a200 con 0x7f37a80b3fd0 2023-10-06T01:41:25.642 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.631+0000 7f37acab5700 1 -- 172.21.15.27:0/3167823609 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37a8064050 con 0x7f37a80b3fd0 2023-10-06T01:41:25.642 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.632+0000 7f379e7fc700 1 -- 172.21.15.27:0/3167823609 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f37a0011bb0 con 0x7f37a80b3fd0 2023-10-06T01:41:25.642 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.632+0000 7f379e7fc700 1 --2- 172.21.15.27:0/3167823609 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3788066f30 0x7f37880693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:25.642 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.632+0000 7f379e7fc700 1 -- 172.21.15.27:0/3167823609 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f37a0085e80 con 0x7f37a80b3fd0 2023-10-06T01:41:25.643 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.633+0000 7f37a5d9b700 1 --2- 172.21.15.27:0/3167823609 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3788066f30 0x7f37880693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:25.643 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.633+0000 7f37a5d9b700 1 --2- 172.21.15.27:0/3167823609 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3788066f30 0x7f37880693e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f37a816d520 tx=0x7f3790008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:25.643 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.637+0000 7f379e7fc700 1 -- 172.21.15.27:0/3167823609 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f37a0052f90 con 0x7f37a80b3fd0 2023-10-06T01:41:25.833 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.831+0000 7f37acab5700 1 -- 172.21.15.27:0/3167823609 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f37a80b1970 con 0x7f37a80b3fd0 2023-10-06T01:41:25.833 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.832+0000 7f379e7fc700 1 -- 172.21.15.27:0/3167823609 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f37a0011e60 con 0x7f37a80b3fd0 2023-10-06T01:41:25.833 INFO:teuthology.orchestra.run.smithi027.stdout:55834574856 2023-10-06T01:41:25.835 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.834+0000 7f37acab5700 1 -- 172.21.15.27:0/3167823609 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3788066f30 msgr2=0x7f37880693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:25.835 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.834+0000 7f37acab5700 1 --2- 172.21.15.27:0/3167823609 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3788066f30 0x7f37880693e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f37a816d520 tx=0x7f3790008040 comp rx=0 tx=0).stop 2023-10-06T01:41:25.835 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.834+0000 7f37acab5700 1 -- 172.21.15.27:0/3167823609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37a80b3fd0 msgr2=0x7f37a8140050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:25.835 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.834+0000 7f37acab5700 1 --2- 172.21.15.27:0/3167823609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37a80b3fd0 0x7f37a8140050 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f37a815deb0 tx=0x7f37a0017000 comp rx=0 tx=0).stop 2023-10-06T01:41:25.835 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.834+0000 7f37acab5700 1 -- 172.21.15.27:0/3167823609 shutdown_connections 2023-10-06T01:41:25.836 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.834+0000 7f37acab5700 1 --2- 172.21.15.27:0/3167823609 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3788066f30 0x7f37880693e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:25.836 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.834+0000 7f37acab5700 1 --2- 172.21.15.27:0/3167823609 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f37a815d8e0 0x7f37a816db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:25.836 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.834+0000 7f37acab5700 1 --2- 172.21.15.27:0/3167823609 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f37a80b3fd0 0x7f37a8140050 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:25.836 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.834+0000 7f37acab5700 1 --2- 172.21.15.27:0/3167823609 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f37a8159500 0x7f37a816d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:25.836 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.834+0000 7f37acab5700 1 -- 172.21.15.27:0/3167823609 >> 172.21.15.27:0/3167823609 conn(0x7f37a8067630 msgr2=0x7f37a80cfc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:25.836 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.834+0000 7f37acab5700 1 -- 172.21.15.27:0/3167823609 shutdown_connections 2023-10-06T01:41:25.836 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.834+0000 7f37acab5700 1 -- 172.21.15.27:0/3167823609 wait complete. 2023-10-06T01:41:25.847 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.2 2023-10-06T01:41:25.847 DEBUG:teuthology.parallel:result is None 2023-10-06T01:41:25.847 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-06T01:41:25.847 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-06T01:41:25.965 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.964+0000 7f5f8ab44700 1 Processor -- start 2023-10-06T01:41:25.965 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.964+0000 7f5f8ab44700 1 -- start start 2023-10-06T01:41:25.968 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.965+0000 7f5f8ab44700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5f8415d480 0x7f5f84069860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:25.968 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.965+0000 7f5f8ab44700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f5f84069da0 0x7f5f840b0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:25.968 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.965+0000 7f5f8ab44700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f840687c0 con 0x7f5f8415d850 2023-10-06T01:41:25.968 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.965+0000 7f5f8ab44700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f84068aa0 con 0x7f5f8415d480 2023-10-06T01:41:25.968 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.965+0000 7f5f8ab44700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f84068d80 con 0x7f5f84069da0 2023-10-06T01:41:25.968 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.965+0000 7f5f89b42700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5f8415d480 0x7f5f84069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:25.968 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.965+0000 7f5f8a343700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5f8415d850 0x7f5f84142a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57024/0 (socket says 172.21.15.27:57024) 2023-10-06T01:41:25.968 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.965+0000 7f5f8a343700 1 -- 172.21.15.27:0/2420906055 learned_addr learned my addr 172.21.15.27:0/2420906055 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:25.968 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.965+0000 7f5f89341700 1 --2- 172.21.15.27:0/2420906055 >> v2:172.21.15.124:3300/0 conn(0x7f5f84069da0 0x7f5f840b0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:25.968 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.965+0000 7f5f89b42700 1 -- 172.21.15.27:0/2420906055 >> v2:172.21.15.124:3300/0 conn(0x7f5f84069da0 msgr2=0x7f5f840b0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:25.968 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.965+0000 7f5f89b42700 1 --2- 172.21.15.27:0/2420906055 >> v2:172.21.15.124:3300/0 conn(0x7f5f84069da0 0x7f5f840b0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:25.969 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.965+0000 7f5f89b42700 1 -- 172.21.15.27:0/2420906055 >> v1:172.21.15.27:6789/0 conn(0x7f5f8415d850 legacy=0x7f5f84142a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:25.969 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.965+0000 7f5f89b42700 1 -- 172.21.15.27:0/2420906055 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f84069060 con 0x7f5f8415d480 2023-10-06T01:41:25.969 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.965+0000 7f5f88b40700 1 -- 172.21.15.27:0/2420906055 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 776280314 0 0) 0x7f5f840687c0 con 0x7f5f8415d850 2023-10-06T01:41:25.969 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.966+0000 7f5f89b42700 1 --2- 172.21.15.27:0/2420906055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f8415d480 0x7f5f84069860 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5f840ce010 tx=0x7f5f7c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=390193bcc90f2968 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:25.969 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.966+0000 7f5f88b40700 1 -- 172.21.15.27:0/2420906055 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c00e070 con 0x7f5f8415d480 2023-10-06T01:41:25.969 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.966+0000 7f5f88b40700 1 -- 172.21.15.27:0/2420906055 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f7c00bb60 con 0x7f5f8415d480 2023-10-06T01:41:25.969 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.966+0000 7f5f88b40700 1 -- 172.21.15.27:0/2420906055 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c012430 con 0x7f5f8415d480 2023-10-06T01:41:25.969 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.966+0000 7f5f8ab44700 1 -- 172.21.15.27:0/2420906055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f8415d480 msgr2=0x7f5f84069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:25.969 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.966+0000 7f5f8ab44700 1 --2- 172.21.15.27:0/2420906055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f8415d480 0x7f5f84069860 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5f840ce010 tx=0x7f5f7c009780 comp rx=0 tx=0).stop 2023-10-06T01:41:25.969 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.966+0000 7f5f8ab44700 1 -- 172.21.15.27:0/2420906055 shutdown_connections 2023-10-06T01:41:25.969 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.966+0000 7f5f8ab44700 1 --2- 172.21.15.27:0/2420906055 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f8415d480 0x7f5f84069860 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:25.969 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.966+0000 7f5f8ab44700 1 --2- 172.21.15.27:0/2420906055 >> v2:172.21.15.124:3300/0 conn(0x7f5f84069da0 0x7f5f840b0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:25.970 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.966+0000 7f5f8ab44700 1 -- 172.21.15.27:0/2420906055 >> 172.21.15.27:0/2420906055 conn(0x7f5f840ccf80 msgr2=0x7f5f840cd380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:25.970 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.966+0000 7f5f8ab44700 1 -- 172.21.15.27:0/2420906055 shutdown_connections 2023-10-06T01:41:25.970 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.966+0000 7f5f8ab44700 1 -- 172.21.15.27:0/2420906055 wait complete. 2023-10-06T01:41:25.972 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.967+0000 7f5f8ab44700 1 Processor -- start 2023-10-06T01:41:25.972 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.968+0000 7f5f8ab44700 1 -- start start 2023-10-06T01:41:25.972 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.968+0000 7f5f8ab44700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f84069da0 0x7f5f84140660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:25.972 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.968+0000 7f5f8ab44700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5f8415d480 0x7f5f840c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:25.972 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.968+0000 7f5f8ab44700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5f8415d850 0x7f5f841698d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:25.973 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.968+0000 7f5f8ab44700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f840c6e70 con 0x7f5f84069da0 2023-10-06T01:41:25.973 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.968+0000 7f5f8ab44700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f840c6280 con 0x7f5f8415d850 2023-10-06T01:41:25.973 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.968+0000 7f5f8ab44700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f840c6540 con 0x7f5f8415d480 2023-10-06T01:41:25.973 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.968+0000 7f5f89b42700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f84069da0 0x7f5f84140660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:25.973 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.968+0000 7f5f89b42700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f84069da0 0x7f5f84140660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52252/0 (socket says 172.21.15.27:52252) 2023-10-06T01:41:25.973 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.968+0000 7f5f89b42700 1 -- 172.21.15.27:0/769746806 learned_addr learned my addr 172.21.15.27:0/769746806 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:25.973 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.968+0000 7f5f8a343700 1 --2- 172.21.15.27:0/769746806 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5f8415d850 0x7f5f841698d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:25.973 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.969+0000 7f5f89341700 1 --2- 172.21.15.27:0/769746806 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5f8415d480 0x7f5f840c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:25.973 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.969+0000 7f5f89b42700 1 -- 172.21.15.27:0/769746806 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5f8415d480 msgr2=0x7f5f840c97f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:25.973 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.969+0000 7f5f89b42700 1 --2- 172.21.15.27:0/769746806 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5f8415d480 0x7f5f840c97f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:25.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.969+0000 7f5f89b42700 1 -- 172.21.15.27:0/769746806 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5f8415d850 msgr2=0x7f5f841698d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:25.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.969+0000 7f5f89b42700 1 --2- 172.21.15.27:0/769746806 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5f8415d850 0x7f5f841698d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:25.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.969+0000 7f5f89b42700 1 -- 172.21.15.27:0/769746806 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f7c009a20 con 0x7f5f84069da0 2023-10-06T01:41:25.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.969+0000 7f5f89b42700 1 --2- 172.21.15.27:0/769746806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f84069da0 0x7f5f84140660 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c009fd0 tx=0x7f5f7c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:25.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.969+0000 7f5f81ffb700 1 -- 172.21.15.27:0/769746806 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c012df0 con 0x7f5f84069da0 2023-10-06T01:41:25.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.969+0000 7f5f81ffb700 1 -- 172.21.15.27:0/769746806 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f7c027470 con 0x7f5f84069da0 2023-10-06T01:41:25.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.969+0000 7f5f8ab44700 1 -- 172.21.15.27:0/769746806 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f840c6820 con 0x7f5f84069da0 2023-10-06T01:41:25.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.969+0000 7f5f81ffb700 1 -- 172.21.15.27:0/769746806 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c027830 con 0x7f5f84069da0 2023-10-06T01:41:25.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.969+0000 7f5f8ab44700 1 -- 172.21.15.27:0/769746806 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f84140ba0 con 0x7f5f84069da0 2023-10-06T01:41:25.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.970+0000 7f5f8ab44700 1 -- 172.21.15.27:0/769746806 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f8404c3c0 con 0x7f5f84069da0 2023-10-06T01:41:25.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.971+0000 7f5f81ffb700 1 -- 172.21.15.27:0/769746806 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5f7c024070 con 0x7f5f84069da0 2023-10-06T01:41:25.975 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.971+0000 7f5f81ffb700 1 --2- 172.21.15.27:0/769746806 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5f70066f30 0x7f5f700693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:25.975 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.971+0000 7f5f81ffb700 1 -- 172.21.15.27:0/769746806 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f5f7c0870c0 con 0x7f5f84069da0 2023-10-06T01:41:25.975 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.971+0000 7f5f89341700 1 --2- 172.21.15.27:0/769746806 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5f70066f30 0x7f5f700693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:25.975 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.972+0000 7f5f89341700 1 --2- 172.21.15.27:0/769746806 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5f70066f30 0x7f5f700693e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5f84140f00 tx=0x7f5f78008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:25.976 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:25.975+0000 7f5f81ffb700 1 -- 172.21.15.27:0/769746806 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5f7c0541d0 con 0x7f5f84069da0 2023-10-06T01:41:26.166 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.164+0000 7f5f8ab44700 1 -- 172.21.15.27:0/769746806 --> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5f840c6fb0 con 0x7f5f70066f30 2023-10-06T01:41:26.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.173+0000 7f5f81ffb700 1 -- 172.21.15.27:0/769746806 <== mgr.4100 v2:172.21.15.124:6832/90278 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286802 (secure 0 0 0) 0x7f5f78009e50 con 0x7f5f70066f30 2023-10-06T01:41:26.178 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:26.181 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.176+0000 7f5f8ab44700 1 -- 172.21.15.27:0/769746806 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5f70066f30 msgr2=0x7f5f700693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:26.181 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.176+0000 7f5f8ab44700 1 --2- 172.21.15.27:0/769746806 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5f70066f30 0x7f5f700693e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5f84140f00 tx=0x7f5f78008040 comp rx=0 tx=0).stop 2023-10-06T01:41:26.182 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.176+0000 7f5f8ab44700 1 -- 172.21.15.27:0/769746806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f84069da0 msgr2=0x7f5f84140660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:26.182 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.176+0000 7f5f8ab44700 1 --2- 172.21.15.27:0/769746806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f84069da0 0x7f5f84140660 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c009fd0 tx=0x7f5f7c019000 comp rx=0 tx=0).stop 2023-10-06T01:41:26.182 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.176+0000 7f5f8ab44700 1 -- 172.21.15.27:0/769746806 shutdown_connections 2023-10-06T01:41:26.182 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.176+0000 7f5f8ab44700 1 --2- 172.21.15.27:0/769746806 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5f70066f30 0x7f5f700693e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:26.183 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.176+0000 7f5f8ab44700 1 --2- 172.21.15.27:0/769746806 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5f8415d850 0x7f5f841698d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:26.183 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.176+0000 7f5f8ab44700 1 --2- 172.21.15.27:0/769746806 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5f84069da0 0x7f5f84140660 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:26.183 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.176+0000 7f5f8ab44700 1 --2- 172.21.15.27:0/769746806 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5f8415d480 0x7f5f840c97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:26.183 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.176+0000 7f5f8ab44700 1 -- 172.21.15.27:0/769746806 >> 172.21.15.27:0/769746806 conn(0x7f5f840ccf80 msgr2=0x7f5f8415bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:26.183 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.176+0000 7f5f8ab44700 1 -- 172.21.15.27:0/769746806 shutdown_connections 2023-10-06T01:41:26.183 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.176+0000 7f5f8ab44700 1 -- 172.21.15.27:0/769746806 wait complete. 2023-10-06T01:41:26.183 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-06T01:41:26.190 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-06T01:41:26.041299+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2381112,"kb_used_data":2808,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752593608,"statfs":{"total":773094113280,"available":770655854592,"internally_reserved":0,"allocated":2875392,"data_stored":1773408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.713509"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.349688+0000","last_change":"2023-10-06T01:41:13.346440+0000","last_active":"2023-10-06T01:41:13.349688+0000","last_peered":"2023-10-06T01:41:13.349688+0000","last_clean":"2023-10-06T01:41:13.349688+0000","last_became_active":"2023-10-06T01:41:13.343345+0000","last_became_peered":"2023-10-06T01:41:13.343345+0000","last_unstale":"2023-10-06T01:41:13.349688+0000","last_undegraded":"2023-10-06T01:41:13.349688+0000","last_fullsized":"2023-10-06T01:41:13.349688+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:15:41.848935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354765+0000","last_change":"2023-10-06T01:41:13.348847+0000","last_active":"2023-10-06T01:41:13.354765+0000","last_peered":"2023-10-06T01:41:13.354765+0000","last_clean":"2023-10-06T01:41:13.354765+0000","last_became_active":"2023-10-06T01:41:13.348585+0000","last_became_peered":"2023-10-06T01:41:13.348585+0000","last_unstale":"2023-10-06T01:41:13.354765+0000","last_undegraded":"2023-10-06T01:41:13.354765+0000","last_fullsized":"2023-10-06T01:41:13.354765+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:31:20.840764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354463+0000","last_change":"2023-10-06T01:41:13.345122+0000","last_active":"2023-10-06T01:41:13.354463+0000","last_peered":"2023-10-06T01:41:13.354463+0000","last_clean":"2023-10-06T01:41:13.354463+0000","last_became_active":"2023-10-06T01:41:13.344752+0000","last_became_peered":"2023-10-06T01:41:13.344752+0000","last_unstale":"2023-10-06T01:41:13.354463+0000","last_undegraded":"2023-10-06T01:41:13.354463+0000","last_fullsized":"2023-10-06T01:41:13.354463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:42:56.041827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.342734+0000","last_change":"2023-10-06T01:41:13.340589+0000","last_active":"2023-10-06T01:41:13.342734+0000","last_peered":"2023-10-06T01:41:13.342734+0000","last_clean":"2023-10-06T01:41:13.342734+0000","last_became_active":"2023-10-06T01:41:13.340339+0000","last_became_peered":"2023-10-06T01:41:13.340339+0000","last_unstale":"2023-10-06T01:41:13.342734+0000","last_undegraded":"2023-10-06T01:41:13.342734+0000","last_fullsized":"2023-10-06T01:41:13.342734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:32:17.938309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.346061+0000","last_change":"2023-10-06T01:41:13.342002+0000","last_active":"2023-10-06T01:41:13.346061+0000","last_peered":"2023-10-06T01:41:13.346061+0000","last_clean":"2023-10-06T01:41:13.346061+0000","last_became_active":"2023-10-06T01:41:13.341403+0000","last_became_peered":"2023-10-06T01:41:13.341403+0000","last_unstale":"2023-10-06T01:41:13.346061+0000","last_undegraded":"2023-10-06T01:41:13.346061+0000","last_fullsized":"2023-10-06T01:41:13.346061+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:34:55.375989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354785+0000","last_change":"2023-10-06T01:41:13.345138+0000","last_active":"2023-10-06T01:41:13.354785+0000","last_peered":"2023-10-06T01:41:13.354785+0000","last_clean":"2023-10-06T01:41:13.354785+0000","last_became_active":"2023-10-06T01:41:13.344810+0000","last_became_peered":"2023-10-06T01:41:13.344810+0000","last_unstale":"2023-10-06T01:41:13.354785+0000","last_undegraded":"2023-10-06T01:41:13.354785+0000","last_fullsized":"2023-10-06T01:41:13.354785+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:17:59.027125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356675+0000","last_change":"2023-10-06T01:41:13.352141+0000","last_active":"2023-10-06T01:41:13.356675+0000","last_peered":"2023-10-06T01:41:13.356675+0000","last_clean":"2023-10-06T01:41:13.356675+0000","last_became_active":"2023-10-06T01:41:13.351628+0000","last_became_peered":"2023-10-06T01:41:13.351628+0000","last_unstale":"2023-10-06T01:41:13.356675+0000","last_undegraded":"2023-10-06T01:41:13.356675+0000","last_fullsized":"2023-10-06T01:41:13.356675+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:03:37.385158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354278+0000","last_change":"2023-10-06T01:41:13.345960+0000","last_active":"2023-10-06T01:41:13.354278+0000","last_peered":"2023-10-06T01:41:13.354278+0000","last_clean":"2023-10-06T01:41:13.354278+0000","last_became_active":"2023-10-06T01:41:13.345702+0000","last_became_peered":"2023-10-06T01:41:13.345702+0000","last_unstale":"2023-10-06T01:41:13.354278+0000","last_undegraded":"2023-10-06T01:41:13.354278+0000","last_fullsized":"2023-10-06T01:41:13.354278+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:03:37.385158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.348168+0000","last_change":"2023-10-06T01:41:13.343792+0000","last_active":"2023-10-06T01:41:13.348168+0000","last_peered":"2023-10-06T01:41:13.348168+0000","last_clean":"2023-10-06T01:41:13.348168+0000","last_became_active":"2023-10-06T01:41:13.343432+0000","last_became_peered":"2023-10-06T01:41:13.343432+0000","last_unstale":"2023-10-06T01:41:13.348168+0000","last_undegraded":"2023-10-06T01:41:13.348168+0000","last_fullsized":"2023-10-06T01:41:13.348168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:52:48.286318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354747+0000","last_change":"2023-10-06T01:41:13.347399+0000","last_active":"2023-10-06T01:41:13.354747+0000","last_peered":"2023-10-06T01:41:13.354747+0000","last_clean":"2023-10-06T01:41:13.354747+0000","last_became_active":"2023-10-06T01:41:13.347136+0000","last_became_peered":"2023-10-06T01:41:13.347136+0000","last_unstale":"2023-10-06T01:41:13.354747+0000","last_undegraded":"2023-10-06T01:41:13.354747+0000","last_fullsized":"2023-10-06T01:41:13.354747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:42:23.199959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354772+0000","last_change":"2023-10-06T01:41:13.350059+0000","last_active":"2023-10-06T01:41:13.354772+0000","last_peered":"2023-10-06T01:41:13.354772+0000","last_clean":"2023-10-06T01:41:13.354772+0000","last_became_active":"2023-10-06T01:41:13.349673+0000","last_became_peered":"2023-10-06T01:41:13.349673+0000","last_unstale":"2023-10-06T01:41:13.354772+0000","last_undegraded":"2023-10-06T01:41:13.354772+0000","last_fullsized":"2023-10-06T01:41:13.354772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:12:57.315612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351538+0000","last_change":"2023-10-06T01:41:13.348001+0000","last_active":"2023-10-06T01:41:13.351538+0000","last_peered":"2023-10-06T01:41:13.351538+0000","last_clean":"2023-10-06T01:41:13.351538+0000","last_became_active":"2023-10-06T01:41:13.347719+0000","last_became_peered":"2023-10-06T01:41:13.347719+0000","last_unstale":"2023-10-06T01:41:13.351538+0000","last_undegraded":"2023-10-06T01:41:13.351538+0000","last_fullsized":"2023-10-06T01:41:13.351538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:37:53.150671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.355181+0000","last_change":"2023-10-06T01:41:13.352875+0000","last_active":"2023-10-06T01:41:13.355181+0000","last_peered":"2023-10-06T01:41:13.355181+0000","last_clean":"2023-10-06T01:41:13.355181+0000","last_became_active":"2023-10-06T01:41:13.352589+0000","last_became_peered":"2023-10-06T01:41:13.352589+0000","last_unstale":"2023-10-06T01:41:13.355181+0000","last_undegraded":"2023-10-06T01:41:13.355181+0000","last_fullsized":"2023-10-06T01:41:13.355181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:49:55.665032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354387+0000","last_change":"2023-10-06T01:41:13.348362+0000","last_active":"2023-10-06T01:41:13.354387+0000","last_peered":"2023-10-06T01:41:13.354387+0000","last_clean":"2023-10-06T01:41:13.354387+0000","last_became_active":"2023-10-06T01:41:13.348181+0000","last_became_peered":"2023-10-06T01:41:13.348181+0000","last_unstale":"2023-10-06T01:41:13.354387+0000","last_undegraded":"2023-10-06T01:41:13.354387+0000","last_fullsized":"2023-10-06T01:41:13.354387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:27:54.074728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356697+0000","last_change":"2023-10-06T01:41:13.352048+0000","last_active":"2023-10-06T01:41:13.356697+0000","last_peered":"2023-10-06T01:41:13.356697+0000","last_clean":"2023-10-06T01:41:13.356697+0000","last_became_active":"2023-10-06T01:41:13.351640+0000","last_became_peered":"2023-10-06T01:41:13.351640+0000","last_unstale":"2023-10-06T01:41:13.356697+0000","last_undegraded":"2023-10-06T01:41:13.356697+0000","last_fullsized":"2023-10-06T01:41:13.356697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:42:23.199959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354602+0000","last_change":"2023-10-06T01:41:13.345677+0000","last_active":"2023-10-06T01:41:13.354602+0000","last_peered":"2023-10-06T01:41:13.354602+0000","last_clean":"2023-10-06T01:41:13.354602+0000","last_became_active":"2023-10-06T01:41:13.345257+0000","last_became_peered":"2023-10-06T01:41:13.345257+0000","last_unstale":"2023-10-06T01:41:13.354602+0000","last_undegraded":"2023-10-06T01:41:13.354602+0000","last_fullsized":"2023-10-06T01:41:13.354602+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:27:07.040029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.348233+0000","last_change":"2023-10-06T01:41:13.344490+0000","last_active":"2023-10-06T01:41:13.348233+0000","last_peered":"2023-10-06T01:41:13.348233+0000","last_clean":"2023-10-06T01:41:13.348233+0000","last_became_active":"2023-10-06T01:41:13.344220+0000","last_became_peered":"2023-10-06T01:41:13.344220+0000","last_unstale":"2023-10-06T01:41:13.348233+0000","last_undegraded":"2023-10-06T01:41:13.348233+0000","last_fullsized":"2023-10-06T01:41:13.348233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T01:51:57.229301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354756+0000","last_change":"2023-10-06T01:41:13.350575+0000","last_active":"2023-10-06T01:41:13.354756+0000","last_peered":"2023-10-06T01:41:13.354756+0000","last_clean":"2023-10-06T01:41:13.354756+0000","last_became_active":"2023-10-06T01:41:13.350311+0000","last_became_peered":"2023-10-06T01:41:13.350311+0000","last_unstale":"2023-10-06T01:41:13.354756+0000","last_undegraded":"2023-10-06T01:41:13.354756+0000","last_fullsized":"2023-10-06T01:41:13.354756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:39:03.091509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351594+0000","last_change":"2023-10-06T01:41:13.347386+0000","last_active":"2023-10-06T01:41:13.351594+0000","last_peered":"2023-10-06T01:41:13.351594+0000","last_clean":"2023-10-06T01:41:13.351594+0000","last_became_active":"2023-10-06T01:41:13.347101+0000","last_became_peered":"2023-10-06T01:41:13.347101+0000","last_unstale":"2023-10-06T01:41:13.351594+0000","last_undegraded":"2023-10-06T01:41:13.351594+0000","last_fullsized":"2023-10-06T01:41:13.351594+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:13:15.011521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351504+0000","last_change":"2023-10-06T01:41:13.342640+0000","last_active":"2023-10-06T01:41:13.351504+0000","last_peered":"2023-10-06T01:41:13.351504+0000","last_clean":"2023-10-06T01:41:13.351504+0000","last_became_active":"2023-10-06T01:41:13.342157+0000","last_became_peered":"2023-10-06T01:41:13.342157+0000","last_unstale":"2023-10-06T01:41:13.351504+0000","last_undegraded":"2023-10-06T01:41:13.351504+0000","last_fullsized":"2023-10-06T01:41:13.351504+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:07:52.521701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356729+0000","last_change":"2023-10-06T01:41:13.352582+0000","last_active":"2023-10-06T01:41:13.356729+0000","last_peered":"2023-10-06T01:41:13.356729+0000","last_clean":"2023-10-06T01:41:13.356729+0000","last_became_active":"2023-10-06T01:41:13.352281+0000","last_became_peered":"2023-10-06T01:41:13.352281+0000","last_unstale":"2023-10-06T01:41:13.356729+0000","last_undegraded":"2023-10-06T01:41:13.356729+0000","last_fullsized":"2023-10-06T01:41:13.356729+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:31:20.840764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351592+0000","last_change":"2023-10-06T01:41:13.345324+0000","last_active":"2023-10-06T01:41:13.351592+0000","last_peered":"2023-10-06T01:41:13.351592+0000","last_clean":"2023-10-06T01:41:13.351592+0000","last_became_active":"2023-10-06T01:41:13.344997+0000","last_became_peered":"2023-10-06T01:41:13.344997+0000","last_unstale":"2023-10-06T01:41:13.351592+0000","last_undegraded":"2023-10-06T01:41:13.351592+0000","last_fullsized":"2023-10-06T01:41:13.351592+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:27:54.074728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.357943+0000","last_change":"2023-10-06T01:41:13.355108+0000","last_active":"2023-10-06T01:41:13.357943+0000","last_peered":"2023-10-06T01:41:13.357943+0000","last_clean":"2023-10-06T01:41:13.357943+0000","last_became_active":"2023-10-06T01:41:13.354621+0000","last_became_peered":"2023-10-06T01:41:13.354621+0000","last_unstale":"2023-10-06T01:41:13.357943+0000","last_undegraded":"2023-10-06T01:41:13.357943+0000","last_fullsized":"2023-10-06T01:41:13.357943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:34:55.375989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356702+0000","last_change":"2023-10-06T01:41:13.352301+0000","last_active":"2023-10-06T01:41:13.356702+0000","last_peered":"2023-10-06T01:41:13.356702+0000","last_clean":"2023-10-06T01:41:13.356702+0000","last_became_active":"2023-10-06T01:41:13.351858+0000","last_became_peered":"2023-10-06T01:41:13.351858+0000","last_unstale":"2023-10-06T01:41:13.356702+0000","last_undegraded":"2023-10-06T01:41:13.356702+0000","last_fullsized":"2023-10-06T01:41:13.356702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:42:46.214805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.348956+0000","last_change":"2023-10-06T01:41:13.344094+0000","last_active":"2023-10-06T01:41:13.348956+0000","last_peered":"2023-10-06T01:41:13.348956+0000","last_clean":"2023-10-06T01:41:13.348956+0000","last_became_active":"2023-10-06T01:41:13.343796+0000","last_became_peered":"2023-10-06T01:41:13.343796+0000","last_unstale":"2023-10-06T01:41:13.348956+0000","last_undegraded":"2023-10-06T01:41:13.348956+0000","last_fullsized":"2023-10-06T01:41:13.348956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:21:30.451862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.355700+0000","last_change":"2023-10-06T01:41:13.350352+0000","last_active":"2023-10-06T01:41:13.355700+0000","last_peered":"2023-10-06T01:41:13.355700+0000","last_clean":"2023-10-06T01:41:13.355700+0000","last_became_active":"2023-10-06T01:41:13.350133+0000","last_became_peered":"2023-10-06T01:41:13.350133+0000","last_unstale":"2023-10-06T01:41:13.355700+0000","last_undegraded":"2023-10-06T01:41:13.355700+0000","last_fullsized":"2023-10-06T01:41:13.355700+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:05:37.961571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354695+0000","last_change":"2023-10-06T01:41:13.353241+0000","last_active":"2023-10-06T01:41:13.354695+0000","last_peered":"2023-10-06T01:41:13.354695+0000","last_clean":"2023-10-06T01:41:13.354695+0000","last_became_active":"2023-10-06T01:41:13.352824+0000","last_became_peered":"2023-10-06T01:41:13.352824+0000","last_unstale":"2023-10-06T01:41:13.354695+0000","last_undegraded":"2023-10-06T01:41:13.354695+0000","last_fullsized":"2023-10-06T01:41:13.354695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:52:48.286318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"19'1","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.348819+0000","last_change":"2023-10-06T01:41:12.339675+0000","last_active":"2023-10-06T01:41:13.348819+0000","last_peered":"2023-10-06T01:41:13.348819+0000","last_clean":"2023-10-06T01:41:13.348819+0000","last_became_active":"2023-10-06T01:41:12.339258+0000","last_became_peered":"2023-10-06T01:41:12.339258+0000","last_unstale":"2023-10-06T01:41:13.348819+0000","last_undegraded":"2023-10-06T01:41:13.348819+0000","last_fullsized":"2023-10-06T01:41:13.348819+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:49:54.658088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.357395+0000","last_change":"2023-10-06T01:41:13.352444+0000","last_active":"2023-10-06T01:41:13.357395+0000","last_peered":"2023-10-06T01:41:13.357395+0000","last_clean":"2023-10-06T01:41:13.357395+0000","last_became_active":"2023-10-06T01:41:13.352134+0000","last_became_peered":"2023-10-06T01:41:13.352134+0000","last_unstale":"2023-10-06T01:41:13.357395+0000","last_undegraded":"2023-10-06T01:41:13.357395+0000","last_fullsized":"2023-10-06T01:41:13.357395+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:11:26.794025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.336930+0000","last_change":"2023-10-06T01:41:12.347654+0000","last_active":"2023-10-06T01:41:13.336930+0000","last_peered":"2023-10-06T01:41:13.336930+0000","last_clean":"2023-10-06T01:41:13.336930+0000","last_became_active":"2023-10-06T01:41:12.347386+0000","last_became_peered":"2023-10-06T01:41:12.347386+0000","last_unstale":"2023-10-06T01:41:13.336930+0000","last_undegraded":"2023-10-06T01:41:13.336930+0000","last_fullsized":"2023-10-06T01:41:13.336930+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:07:30.529679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356475+0000","last_change":"2023-10-06T01:41:13.352350+0000","last_active":"2023-10-06T01:41:13.356475+0000","last_peered":"2023-10-06T01:41:13.356475+0000","last_clean":"2023-10-06T01:41:13.356475+0000","last_became_active":"2023-10-06T01:41:13.352091+0000","last_became_peered":"2023-10-06T01:41:13.352091+0000","last_unstale":"2023-10-06T01:41:13.356475+0000","last_undegraded":"2023-10-06T01:41:13.356475+0000","last_fullsized":"2023-10-06T01:41:13.356475+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:31:20.840764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.334517+0000","last_change":"2023-10-06T01:41:12.343086+0000","last_active":"2023-10-06T01:41:13.334517+0000","last_peered":"2023-10-06T01:41:13.334517+0000","last_clean":"2023-10-06T01:41:13.334517+0000","last_became_active":"2023-10-06T01:41:12.342595+0000","last_became_peered":"2023-10-06T01:41:12.342595+0000","last_unstale":"2023-10-06T01:41:13.334517+0000","last_undegraded":"2023-10-06T01:41:13.334517+0000","last_fullsized":"2023-10-06T01:41:13.334517+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:15:40.841990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351933+0000","last_change":"2023-10-06T01:41:13.347027+0000","last_active":"2023-10-06T01:41:13.351933+0000","last_peered":"2023-10-06T01:41:13.351933+0000","last_clean":"2023-10-06T01:41:13.351933+0000","last_became_active":"2023-10-06T01:41:13.346469+0000","last_became_peered":"2023-10-06T01:41:13.346469+0000","last_unstale":"2023-10-06T01:41:13.351933+0000","last_undegraded":"2023-10-06T01:41:13.351933+0000","last_fullsized":"2023-10-06T01:41:13.351933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:37:14.679660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.329704+0000","last_change":"2023-10-06T01:41:12.341922+0000","last_active":"2023-10-06T01:41:13.329704+0000","last_peered":"2023-10-06T01:41:13.329704+0000","last_clean":"2023-10-06T01:41:13.329704+0000","last_became_active":"2023-10-06T01:41:12.341543+0000","last_became_peered":"2023-10-06T01:41:12.341543+0000","last_unstale":"2023-10-06T01:41:13.329704+0000","last_undegraded":"2023-10-06T01:41:13.329704+0000","last_fullsized":"2023-10-06T01:41:13.329704+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:13:14.004577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352198+0000","last_change":"2023-10-06T01:41:13.345274+0000","last_active":"2023-10-06T01:41:13.352198+0000","last_peered":"2023-10-06T01:41:13.352198+0000","last_clean":"2023-10-06T01:41:13.352198+0000","last_became_active":"2023-10-06T01:41:13.344992+0000","last_became_peered":"2023-10-06T01:41:13.344992+0000","last_unstale":"2023-10-06T01:41:13.352198+0000","last_undegraded":"2023-10-06T01:41:13.352198+0000","last_fullsized":"2023-10-06T01:41:13.352198+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:39:03.091509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.339176+0000","last_change":"2023-10-06T01:41:12.343531+0000","last_active":"2023-10-06T01:41:13.339176+0000","last_peered":"2023-10-06T01:41:13.339176+0000","last_clean":"2023-10-06T01:41:13.339176+0000","last_became_active":"2023-10-06T01:41:12.343256+0000","last_became_peered":"2023-10-06T01:41:12.343256+0000","last_unstale":"2023-10-06T01:41:13.339176+0000","last_undegraded":"2023-10-06T01:41:13.339176+0000","last_fullsized":"2023-10-06T01:41:13.339176+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:16:14.826869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.358071+0000","last_change":"2023-10-06T01:41:13.357361+0000","last_active":"2023-10-06T01:41:13.358071+0000","last_peered":"2023-10-06T01:41:13.358071+0000","last_clean":"2023-10-06T01:41:13.358071+0000","last_became_active":"2023-10-06T01:41:13.356980+0000","last_became_peered":"2023-10-06T01:41:13.356980+0000","last_unstale":"2023-10-06T01:41:13.358071+0000","last_undegraded":"2023-10-06T01:41:13.358071+0000","last_fullsized":"2023-10-06T01:41:13.358071+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:42:51.581939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"19'3","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.355632+0000","last_change":"2023-10-06T01:41:12.350010+0000","last_active":"2023-10-06T01:41:13.355632+0000","last_peered":"2023-10-06T01:41:13.355632+0000","last_clean":"2023-10-06T01:41:13.355632+0000","last_became_active":"2023-10-06T01:41:12.349758+0000","last_became_peered":"2023-10-06T01:41:12.349758+0000","last_unstale":"2023-10-06T01:41:13.355632+0000","last_undegraded":"2023-10-06T01:41:13.355632+0000","last_fullsized":"2023-10-06T01:41:13.355632+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:12:51.734395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352993+0000","last_change":"2023-10-06T01:41:13.348953+0000","last_active":"2023-10-06T01:41:13.352993+0000","last_peered":"2023-10-06T01:41:13.352993+0000","last_clean":"2023-10-06T01:41:13.352993+0000","last_became_active":"2023-10-06T01:41:13.348683+0000","last_became_peered":"2023-10-06T01:41:13.348683+0000","last_unstale":"2023-10-06T01:41:13.352993+0000","last_undegraded":"2023-10-06T01:41:13.352993+0000","last_fullsized":"2023-10-06T01:41:13.352993+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:42:46.214805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"19'6","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.357074+0000","last_change":"2023-10-06T01:41:12.340922+0000","last_active":"2023-10-06T01:41:13.357074+0000","last_peered":"2023-10-06T01:41:13.357074+0000","last_clean":"2023-10-06T01:41:13.357074+0000","last_became_active":"2023-10-06T01:41:12.340598+0000","last_became_peered":"2023-10-06T01:41:12.340598+0000","last_unstale":"2023-10-06T01:41:13.357074+0000","last_undegraded":"2023-10-06T01:41:13.357074+0000","last_fullsized":"2023-10-06T01:41:13.357074+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:06:19.029137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351653+0000","last_change":"2023-10-06T01:41:13.343918+0000","last_active":"2023-10-06T01:41:13.351653+0000","last_peered":"2023-10-06T01:41:13.351653+0000","last_clean":"2023-10-06T01:41:13.351653+0000","last_became_active":"2023-10-06T01:41:13.343621+0000","last_became_peered":"2023-10-06T01:41:13.343621+0000","last_unstale":"2023-10-06T01:41:13.351653+0000","last_undegraded":"2023-10-06T01:41:13.351653+0000","last_fullsized":"2023-10-06T01:41:13.351653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:42:44.799267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.350439+0000","last_change":"2023-10-06T01:41:12.343541+0000","last_active":"2023-10-06T01:41:13.350439+0000","last_peered":"2023-10-06T01:41:13.350439+0000","last_clean":"2023-10-06T01:41:13.350439+0000","last_became_active":"2023-10-06T01:41:12.342645+0000","last_became_peered":"2023-10-06T01:41:12.342645+0000","last_unstale":"2023-10-06T01:41:13.350439+0000","last_undegraded":"2023-10-06T01:41:13.350439+0000","last_fullsized":"2023-10-06T01:41:13.350439+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:16:14.826869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.357594+0000","last_change":"2023-10-06T01:41:13.352574+0000","last_active":"2023-10-06T01:41:13.357594+0000","last_peered":"2023-10-06T01:41:13.357594+0000","last_clean":"2023-10-06T01:41:13.357594+0000","last_became_active":"2023-10-06T01:41:13.352300+0000","last_became_peered":"2023-10-06T01:41:13.352300+0000","last_unstale":"2023-10-06T01:41:13.357594+0000","last_undegraded":"2023-10-06T01:41:13.357594+0000","last_fullsized":"2023-10-06T01:41:13.357594+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:27:54.074728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.329224+0000","last_change":"2023-10-06T01:41:12.344679+0000","last_active":"2023-10-06T01:41:13.329224+0000","last_peered":"2023-10-06T01:41:13.329224+0000","last_clean":"2023-10-06T01:41:13.329224+0000","last_became_active":"2023-10-06T01:41:12.344430+0000","last_became_peered":"2023-10-06T01:41:12.344430+0000","last_unstale":"2023-10-06T01:41:13.329224+0000","last_undegraded":"2023-10-06T01:41:13.329224+0000","last_fullsized":"2023-10-06T01:41:13.329224+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:27:01.582307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.360786+0000","last_change":"2023-10-06T01:41:13.357148+0000","last_active":"2023-10-06T01:41:13.360786+0000","last_peered":"2023-10-06T01:41:13.360786+0000","last_clean":"2023-10-06T01:41:13.360786+0000","last_became_active":"2023-10-06T01:41:13.356843+0000","last_became_peered":"2023-10-06T01:41:13.356843+0000","last_unstale":"2023-10-06T01:41:13.360786+0000","last_undegraded":"2023-10-06T01:41:13.360786+0000","last_fullsized":"2023-10-06T01:41:13.360786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T01:51:57.229301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.334167+0000","last_change":"2023-10-06T01:41:12.344068+0000","last_active":"2023-10-06T01:41:13.334167+0000","last_peered":"2023-10-06T01:41:13.334167+0000","last_clean":"2023-10-06T01:41:13.334167+0000","last_became_active":"2023-10-06T01:41:12.342562+0000","last_became_peered":"2023-10-06T01:41:12.342562+0000","last_unstale":"2023-10-06T01:41:13.334167+0000","last_undegraded":"2023-10-06T01:41:13.334167+0000","last_fullsized":"2023-10-06T01:41:13.334167+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:37:13.672716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.346441+0000","last_change":"2023-10-06T01:41:13.343508+0000","last_active":"2023-10-06T01:41:13.346441+0000","last_peered":"2023-10-06T01:41:13.346441+0000","last_clean":"2023-10-06T01:41:13.346441+0000","last_became_active":"2023-10-06T01:41:13.343249+0000","last_became_peered":"2023-10-06T01:41:13.343249+0000","last_unstale":"2023-10-06T01:41:13.346441+0000","last_undegraded":"2023-10-06T01:41:13.346441+0000","last_fullsized":"2023-10-06T01:41:13.346441+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:03:37.385158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"19'8","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.347028+0000","last_change":"2023-10-06T01:41:12.342949+0000","last_active":"2023-10-06T01:41:13.347028+0000","last_peered":"2023-10-06T01:41:13.347028+0000","last_clean":"2023-10-06T01:41:13.347028+0000","last_became_active":"2023-10-06T01:41:12.342687+0000","last_became_peered":"2023-10-06T01:41:12.342687+0000","last_unstale":"2023-10-06T01:41:13.347028+0000","last_undegraded":"2023-10-06T01:41:13.347028+0000","last_fullsized":"2023-10-06T01:41:13.347028+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:37:52.143727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.357895+0000","last_change":"2023-10-06T01:41:13.354861+0000","last_active":"2023-10-06T01:41:13.357895+0000","last_peered":"2023-10-06T01:41:13.357895+0000","last_clean":"2023-10-06T01:41:13.357895+0000","last_became_active":"2023-10-06T01:41:13.354190+0000","last_became_peered":"2023-10-06T01:41:13.354190+0000","last_unstale":"2023-10-06T01:41:13.357895+0000","last_undegraded":"2023-10-06T01:41:13.357895+0000","last_fullsized":"2023-10-06T01:41:13.357895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:45:07.571069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"19'1","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.349986+0000","last_change":"2023-10-06T01:41:12.344048+0000","last_active":"2023-10-06T01:41:13.349986+0000","last_peered":"2023-10-06T01:41:13.349986+0000","last_clean":"2023-10-06T01:41:13.349986+0000","last_became_active":"2023-10-06T01:41:12.343532+0000","last_became_peered":"2023-10-06T01:41:12.343532+0000","last_unstale":"2023-10-06T01:41:13.349986+0000","last_undegraded":"2023-10-06T01:41:13.349986+0000","last_fullsized":"2023-10-06T01:41:13.349986+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:50:44.552768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354457+0000","last_change":"2023-10-06T01:41:13.351276+0000","last_active":"2023-10-06T01:41:13.354457+0000","last_peered":"2023-10-06T01:41:13.354457+0000","last_clean":"2023-10-06T01:41:13.354457+0000","last_became_active":"2023-10-06T01:41:13.351005+0000","last_became_peered":"2023-10-06T01:41:13.351005+0000","last_unstale":"2023-10-06T01:41:13.354457+0000","last_undegraded":"2023-10-06T01:41:13.354457+0000","last_fullsized":"2023-10-06T01:41:13.354457+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:42:51.581939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356755+0000","last_change":"2023-10-06T01:41:12.344159+0000","last_active":"2023-10-06T01:41:13.356755+0000","last_peered":"2023-10-06T01:41:13.356755+0000","last_clean":"2023-10-06T01:41:13.356755+0000","last_became_active":"2023-10-06T01:41:12.343593+0000","last_became_peered":"2023-10-06T01:41:12.343593+0000","last_unstale":"2023-10-06T01:41:13.356755+0000","last_undegraded":"2023-10-06T01:41:13.356755+0000","last_fullsized":"2023-10-06T01:41:13.356755+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:27:01.582307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354635+0000","last_change":"2023-10-06T01:41:13.349120+0000","last_active":"2023-10-06T01:41:13.354635+0000","last_peered":"2023-10-06T01:41:13.354635+0000","last_clean":"2023-10-06T01:41:13.354635+0000","last_became_active":"2023-10-06T01:41:13.348850+0000","last_became_peered":"2023-10-06T01:41:13.348850+0000","last_unstale":"2023-10-06T01:41:13.354635+0000","last_undegraded":"2023-10-06T01:41:13.354635+0000","last_fullsized":"2023-10-06T01:41:13.354635+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:27:32.273170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"19'1","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.358476+0000","last_change":"2023-10-06T01:41:12.344880+0000","last_active":"2023-10-06T01:41:13.358476+0000","last_peered":"2023-10-06T01:41:13.358476+0000","last_clean":"2023-10-06T01:41:13.358476+0000","last_became_active":"2023-10-06T01:41:12.344676+0000","last_became_peered":"2023-10-06T01:41:12.344676+0000","last_unstale":"2023-10-06T01:41:13.358476+0000","last_undegraded":"2023-10-06T01:41:13.358476+0000","last_fullsized":"2023-10-06T01:41:13.358476+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:21:09.833061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.353951+0000","last_change":"2023-10-06T01:41:13.348117+0000","last_active":"2023-10-06T01:41:13.353951+0000","last_peered":"2023-10-06T01:41:13.353951+0000","last_clean":"2023-10-06T01:41:13.353951+0000","last_became_active":"2023-10-06T01:41:13.347872+0000","last_became_peered":"2023-10-06T01:41:13.347872+0000","last_unstale":"2023-10-06T01:41:13.353951+0000","last_undegraded":"2023-10-06T01:41:13.353951+0000","last_fullsized":"2023-10-06T01:41:13.353951+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:21:10.840006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.338677+0000","last_change":"2023-10-06T01:41:12.341362+0000","last_active":"2023-10-06T01:41:13.338677+0000","last_peered":"2023-10-06T01:41:13.338677+0000","last_clean":"2023-10-06T01:41:13.338677+0000","last_became_active":"2023-10-06T01:41:12.340855+0000","last_became_peered":"2023-10-06T01:41:12.340855+0000","last_unstale":"2023-10-06T01:41:13.338677+0000","last_undegraded":"2023-10-06T01:41:13.338677+0000","last_fullsized":"2023-10-06T01:41:13.338677+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:15:40.841990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.349189+0000","last_change":"2023-10-06T01:41:13.344966+0000","last_active":"2023-10-06T01:41:13.349189+0000","last_peered":"2023-10-06T01:41:13.349189+0000","last_clean":"2023-10-06T01:41:13.349189+0000","last_became_active":"2023-10-06T01:41:13.344700+0000","last_became_peered":"2023-10-06T01:41:13.344700+0000","last_unstale":"2023-10-06T01:41:13.349189+0000","last_undegraded":"2023-10-06T01:41:13.349189+0000","last_fullsized":"2023-10-06T01:41:13.349189+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:42:23.199959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.346337+0000","last_change":"2023-10-06T01:41:12.343169+0000","last_active":"2023-10-06T01:41:13.346337+0000","last_peered":"2023-10-06T01:41:13.346337+0000","last_clean":"2023-10-06T01:41:13.346337+0000","last_became_active":"2023-10-06T01:41:12.342474+0000","last_became_peered":"2023-10-06T01:41:12.342474+0000","last_unstale":"2023-10-06T01:41:13.346337+0000","last_undegraded":"2023-10-06T01:41:13.346337+0000","last_fullsized":"2023-10-06T01:41:13.346337+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:27:01.582307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351608+0000","last_change":"2023-10-06T01:41:13.343671+0000","last_active":"2023-10-06T01:41:13.351608+0000","last_peered":"2023-10-06T01:41:13.351608+0000","last_clean":"2023-10-06T01:41:13.351608+0000","last_became_active":"2023-10-06T01:41:13.341868+0000","last_became_peered":"2023-10-06T01:41:13.341868+0000","last_unstale":"2023-10-06T01:41:13.351608+0000","last_undegraded":"2023-10-06T01:41:13.351608+0000","last_fullsized":"2023-10-06T01:41:13.351608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T01:51:57.229301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.329621+0000","last_change":"2023-10-06T01:41:12.339878+0000","last_active":"2023-10-06T01:41:13.329621+0000","last_peered":"2023-10-06T01:41:13.329621+0000","last_clean":"2023-10-06T01:41:13.329621+0000","last_became_active":"2023-10-06T01:41:12.338935+0000","last_became_peered":"2023-10-06T01:41:12.338935+0000","last_unstale":"2023-10-06T01:41:13.329621+0000","last_undegraded":"2023-10-06T01:41:13.329621+0000","last_fullsized":"2023-10-06T01:41:13.329621+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:44:53.014481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351640+0000","last_change":"2023-10-06T01:41:13.344292+0000","last_active":"2023-10-06T01:41:13.351640+0000","last_peered":"2023-10-06T01:41:13.351640+0000","last_clean":"2023-10-06T01:41:13.351640+0000","last_became_active":"2023-10-06T01:41:13.343947+0000","last_became_peered":"2023-10-06T01:41:13.343947+0000","last_unstale":"2023-10-06T01:41:13.351640+0000","last_undegraded":"2023-10-06T01:41:13.351640+0000","last_fullsized":"2023-10-06T01:41:13.351640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:45:07.571069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.331713+0000","last_change":"2023-10-06T01:41:12.336355+0000","last_active":"2023-10-06T01:41:13.331713+0000","last_peered":"2023-10-06T01:41:13.331713+0000","last_clean":"2023-10-06T01:41:13.331713+0000","last_became_active":"2023-10-06T01:41:12.336064+0000","last_became_peered":"2023-10-06T01:41:12.336064+0000","last_unstale":"2023-10-06T01:41:13.331713+0000","last_undegraded":"2023-10-06T01:41:13.331713+0000","last_fullsized":"2023-10-06T01:41:13.331713+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:16:14.826869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354641+0000","last_change":"2023-10-06T01:41:13.346404+0000","last_active":"2023-10-06T01:41:13.354641+0000","last_peered":"2023-10-06T01:41:13.354641+0000","last_clean":"2023-10-06T01:41:13.354641+0000","last_became_active":"2023-10-06T01:41:13.343805+0000","last_became_peered":"2023-10-06T01:41:13.343805+0000","last_unstale":"2023-10-06T01:41:13.354641+0000","last_undegraded":"2023-10-06T01:41:13.354641+0000","last_fullsized":"2023-10-06T01:41:13.354641+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:11:26.794025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.334388+0000","last_change":"2023-10-06T01:41:12.345661+0000","last_active":"2023-10-06T01:41:13.334388+0000","last_peered":"2023-10-06T01:41:13.334388+0000","last_clean":"2023-10-06T01:41:13.334388+0000","last_became_active":"2023-10-06T01:41:12.345363+0000","last_became_peered":"2023-10-06T01:41:12.345363+0000","last_unstale":"2023-10-06T01:41:13.334388+0000","last_undegraded":"2023-10-06T01:41:13.334388+0000","last_fullsized":"2023-10-06T01:41:13.334388+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:50:44.552768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356650+0000","last_change":"2023-10-06T01:41:13.353633+0000","last_active":"2023-10-06T01:41:13.356650+0000","last_peered":"2023-10-06T01:41:13.356650+0000","last_clean":"2023-10-06T01:41:13.356650+0000","last_became_active":"2023-10-06T01:41:13.353318+0000","last_became_peered":"2023-10-06T01:41:13.353318+0000","last_unstale":"2023-10-06T01:41:13.356650+0000","last_undegraded":"2023-10-06T01:41:13.356650+0000","last_fullsized":"2023-10-06T01:41:13.356650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:42:23.199959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356615+0000","last_change":"2023-10-06T01:41:12.340463+0000","last_active":"2023-10-06T01:41:13.356615+0000","last_peered":"2023-10-06T01:41:13.356615+0000","last_clean":"2023-10-06T01:41:13.356615+0000","last_became_active":"2023-10-06T01:41:12.340186+0000","last_became_peered":"2023-10-06T01:41:12.340186+0000","last_unstale":"2023-10-06T01:41:13.356615+0000","last_undegraded":"2023-10-06T01:41:13.356615+0000","last_fullsized":"2023-10-06T01:41:13.356615+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:16:14.826869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.330869+0000","last_change":"2023-10-06T01:41:07.454295+0000","last_active":"2023-10-06T01:41:13.330869+0000","last_peered":"2023-10-06T01:41:13.330869+0000","last_clean":"2023-10-06T01:41:13.330869+0000","last_became_active":"2023-10-06T01:41:07.454034+0000","last_became_peered":"2023-10-06T01:41:07.454034+0000","last_unstale":"2023-10-06T01:41:13.330869+0000","last_undegraded":"2023-10-06T01:41:13.330869+0000","last_fullsized":"2023-10-06T01:41:13.330869+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:32:58.914183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354272+0000","last_change":"2023-10-06T01:41:13.351289+0000","last_active":"2023-10-06T01:41:13.354272+0000","last_peered":"2023-10-06T01:41:13.354272+0000","last_clean":"2023-10-06T01:41:13.354272+0000","last_became_active":"2023-10-06T01:41:13.351022+0000","last_became_peered":"2023-10-06T01:41:13.351022+0000","last_unstale":"2023-10-06T01:41:13.354272+0000","last_undegraded":"2023-10-06T01:41:13.354272+0000","last_fullsized":"2023-10-06T01:41:13.354272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T01:51:57.229301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.334655+0000","last_change":"2023-10-06T01:41:07.261955+0000","last_active":"2023-10-06T01:41:13.334655+0000","last_peered":"2023-10-06T01:41:13.334655+0000","last_clean":"2023-10-06T01:41:13.334655+0000","last_became_active":"2023-10-06T01:41:07.261692+0000","last_became_peered":"2023-10-06T01:41:07.261692+0000","last_unstale":"2023-10-06T01:41:13.334655+0000","last_undegraded":"2023-10-06T01:41:13.334655+0000","last_fullsized":"2023-10-06T01:41:13.334655+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"20'1","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-06T01:41:15.784088+0000","last_change":"2023-10-06T01:41:12.341312+0000","last_active":"2023-10-06T01:41:15.784088+0000","last_peered":"2023-10-06T01:41:15.784088+0000","last_clean":"2023-10-06T01:41:15.784088+0000","last_became_active":"2023-10-06T01:41:12.340997+0000","last_became_peered":"2023-10-06T01:41:12.340997+0000","last_unstale":"2023-10-06T01:41:15.784088+0000","last_undegraded":"2023-10-06T01:41:15.784088+0000","last_fullsized":"2023-10-06T01:41:15.784088+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:06:19.029137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351571+0000","last_change":"2023-10-06T01:41:13.344156+0000","last_active":"2023-10-06T01:41:13.351571+0000","last_peered":"2023-10-06T01:41:13.351571+0000","last_clean":"2023-10-06T01:41:13.351571+0000","last_became_active":"2023-10-06T01:41:13.343601+0000","last_became_peered":"2023-10-06T01:41:13.343601+0000","last_unstale":"2023-10-06T01:41:13.351571+0000","last_undegraded":"2023-10-06T01:41:13.351571+0000","last_fullsized":"2023-10-06T01:41:13.351571+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:21:42.851681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.336375+0000","last_change":"2023-10-06T01:41:07.258362+0000","last_active":"2023-10-06T01:41:13.336375+0000","last_peered":"2023-10-06T01:41:13.336375+0000","last_clean":"2023-10-06T01:41:13.336375+0000","last_became_active":"2023-10-06T01:41:07.258055+0000","last_became_peered":"2023-10-06T01:41:07.258055+0000","last_unstale":"2023-10-06T01:41:13.336375+0000","last_undegraded":"2023-10-06T01:41:13.336375+0000","last_fullsized":"2023-10-06T01:41:13.336375+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"16'117","reported_seq":163,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.334859+0000","last_change":"2023-10-06T01:41:07.261164+0000","last_active":"2023-10-06T01:41:13.334859+0000","last_peered":"2023-10-06T01:41:13.334859+0000","last_clean":"2023-10-06T01:41:13.334859+0000","last_became_active":"2023-10-06T01:41:07.260896+0000","last_became_peered":"2023-10-06T01:41:07.260896+0000","last_unstale":"2023-10-06T01:41:13.334859+0000","last_undegraded":"2023-10-06T01:41:13.334859+0000","last_fullsized":"2023-10-06T01:41:13.334859+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:05:22.934401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.328964+0000","last_change":"2023-10-06T01:41:12.343057+0000","last_active":"2023-10-06T01:41:13.328964+0000","last_peered":"2023-10-06T01:41:13.328964+0000","last_clean":"2023-10-06T01:41:13.328964+0000","last_became_active":"2023-10-06T01:41:12.342793+0000","last_became_peered":"2023-10-06T01:41:12.342793+0000","last_unstale":"2023-10-06T01:41:13.328964+0000","last_undegraded":"2023-10-06T01:41:13.328964+0000","last_fullsized":"2023-10-06T01:41:13.328964+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:15:40.841990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351969+0000","last_change":"2023-10-06T01:41:13.346571+0000","last_active":"2023-10-06T01:41:13.351969+0000","last_peered":"2023-10-06T01:41:13.351969+0000","last_clean":"2023-10-06T01:41:13.351969+0000","last_became_active":"2023-10-06T01:41:13.346077+0000","last_became_peered":"2023-10-06T01:41:13.346077+0000","last_unstale":"2023-10-06T01:41:13.351969+0000","last_undegraded":"2023-10-06T01:41:13.351969+0000","last_fullsized":"2023-10-06T01:41:13.351969+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:42:51.581939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.333951+0000","last_change":"2023-10-06T01:41:07.456511+0000","last_active":"2023-10-06T01:41:13.333951+0000","last_peered":"2023-10-06T01:41:13.333951+0000","last_clean":"2023-10-06T01:41:13.333951+0000","last_became_active":"2023-10-06T01:41:07.456024+0000","last_became_peered":"2023-10-06T01:41:07.456024+0000","last_unstale":"2023-10-06T01:41:13.333951+0000","last_undegraded":"2023-10-06T01:41:13.333951+0000","last_fullsized":"2023-10-06T01:41:13.333951+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:51:02.949231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.336635+0000","last_change":"2023-10-06T01:41:12.343382+0000","last_active":"2023-10-06T01:41:13.336635+0000","last_peered":"2023-10-06T01:41:13.336635+0000","last_clean":"2023-10-06T01:41:13.336635+0000","last_became_active":"2023-10-06T01:41:12.343088+0000","last_became_peered":"2023-10-06T01:41:12.343088+0000","last_unstale":"2023-10-06T01:41:13.336635+0000","last_undegraded":"2023-10-06T01:41:13.336635+0000","last_fullsized":"2023-10-06T01:41:13.336635+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:44:53.014481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.350157+0000","last_change":"2023-10-06T01:41:13.346852+0000","last_active":"2023-10-06T01:41:13.350157+0000","last_peered":"2023-10-06T01:41:13.350157+0000","last_clean":"2023-10-06T01:41:13.350157+0000","last_became_active":"2023-10-06T01:41:13.346571+0000","last_became_peered":"2023-10-06T01:41:13.346571+0000","last_unstale":"2023-10-06T01:41:13.350157+0000","last_undegraded":"2023-10-06T01:41:13.350157+0000","last_fullsized":"2023-10-06T01:41:13.350157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:25:04.563235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.334453+0000","last_change":"2023-10-06T01:41:07.456273+0000","last_active":"2023-10-06T01:41:13.334453+0000","last_peered":"2023-10-06T01:41:13.334453+0000","last_clean":"2023-10-06T01:41:13.334453+0000","last_became_active":"2023-10-06T01:41:07.456000+0000","last_became_peered":"2023-10-06T01:41:07.456000+0000","last_unstale":"2023-10-06T01:41:13.334453+0000","last_undegraded":"2023-10-06T01:41:13.334453+0000","last_fullsized":"2023-10-06T01:41:13.334453+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:01:42.633388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.334637+0000","last_change":"2023-10-06T01:41:12.342215+0000","last_active":"2023-10-06T01:41:13.334637+0000","last_peered":"2023-10-06T01:41:13.334637+0000","last_clean":"2023-10-06T01:41:13.334637+0000","last_became_active":"2023-10-06T01:41:12.341493+0000","last_became_peered":"2023-10-06T01:41:12.341493+0000","last_unstale":"2023-10-06T01:41:13.334637+0000","last_undegraded":"2023-10-06T01:41:13.334637+0000","last_fullsized":"2023-10-06T01:41:13.334637+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:49:54.658088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352134+0000","last_change":"2023-10-06T01:41:13.346680+0000","last_active":"2023-10-06T01:41:13.352134+0000","last_peered":"2023-10-06T01:41:13.352134+0000","last_clean":"2023-10-06T01:41:13.352134+0000","last_became_active":"2023-10-06T01:41:13.346326+0000","last_became_peered":"2023-10-06T01:41:13.346326+0000","last_unstale":"2023-10-06T01:41:13.352134+0000","last_undegraded":"2023-10-06T01:41:13.352134+0000","last_fullsized":"2023-10-06T01:41:13.352134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:12:52.741339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.338955+0000","last_change":"2023-10-06T01:41:07.456688+0000","last_active":"2023-10-06T01:41:13.338955+0000","last_peered":"2023-10-06T01:41:13.338955+0000","last_clean":"2023-10-06T01:41:13.338955+0000","last_became_active":"2023-10-06T01:41:07.456304+0000","last_became_peered":"2023-10-06T01:41:07.456304+0000","last_unstale":"2023-10-06T01:41:13.338955+0000","last_undegraded":"2023-10-06T01:41:13.338955+0000","last_fullsized":"2023-10-06T01:41:13.338955+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.332203+0000","last_change":"2023-10-06T01:41:12.342354+0000","last_active":"2023-10-06T01:41:13.332203+0000","last_peered":"2023-10-06T01:41:13.332203+0000","last_clean":"2023-10-06T01:41:13.332203+0000","last_became_active":"2023-10-06T01:41:12.342066+0000","last_became_peered":"2023-10-06T01:41:12.342066+0000","last_unstale":"2023-10-06T01:41:13.332203+0000","last_undegraded":"2023-10-06T01:41:13.332203+0000","last_fullsized":"2023-10-06T01:41:13.332203+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:15:40.841990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352905+0000","last_change":"2023-10-06T01:41:13.348241+0000","last_active":"2023-10-06T01:41:13.352905+0000","last_peered":"2023-10-06T01:41:13.352905+0000","last_clean":"2023-10-06T01:41:13.352905+0000","last_became_active":"2023-10-06T01:41:13.347979+0000","last_became_peered":"2023-10-06T01:41:13.347979+0000","last_unstale":"2023-10-06T01:41:13.352905+0000","last_undegraded":"2023-10-06T01:41:13.352905+0000","last_fullsized":"2023-10-06T01:41:13.352905+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:41:59.186469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.329416+0000","last_change":"2023-10-06T01:41:07.456388+0000","last_active":"2023-10-06T01:41:13.329416+0000","last_peered":"2023-10-06T01:41:13.329416+0000","last_clean":"2023-10-06T01:41:13.329416+0000","last_became_active":"2023-10-06T01:41:07.456045+0000","last_became_peered":"2023-10-06T01:41:07.456045+0000","last_unstale":"2023-10-06T01:41:13.329416+0000","last_undegraded":"2023-10-06T01:41:13.329416+0000","last_fullsized":"2023-10-06T01:41:13.329416+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.338387+0000","last_change":"2023-10-06T01:41:12.344268+0000","last_active":"2023-10-06T01:41:13.338387+0000","last_peered":"2023-10-06T01:41:13.338387+0000","last_clean":"2023-10-06T01:41:13.338387+0000","last_became_active":"2023-10-06T01:41:12.343996+0000","last_became_peered":"2023-10-06T01:41:12.343996+0000","last_unstale":"2023-10-06T01:41:13.338387+0000","last_undegraded":"2023-10-06T01:41:13.338387+0000","last_fullsized":"2023-10-06T01:41:13.338387+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:37:52.143727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351761+0000","last_change":"2023-10-06T01:41:13.346184+0000","last_active":"2023-10-06T01:41:13.351761+0000","last_peered":"2023-10-06T01:41:13.351761+0000","last_clean":"2023-10-06T01:41:13.351761+0000","last_became_active":"2023-10-06T01:41:13.345570+0000","last_became_peered":"2023-10-06T01:41:13.345570+0000","last_unstale":"2023-10-06T01:41:13.351761+0000","last_undegraded":"2023-10-06T01:41:13.351761+0000","last_fullsized":"2023-10-06T01:41:13.351761+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:42:46.214805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.330470+0000","last_change":"2023-10-06T01:41:07.258162+0000","last_active":"2023-10-06T01:41:13.330470+0000","last_peered":"2023-10-06T01:41:13.330470+0000","last_clean":"2023-10-06T01:41:13.330470+0000","last_became_active":"2023-10-06T01:41:07.257795+0000","last_became_peered":"2023-10-06T01:41:07.257795+0000","last_unstale":"2023-10-06T01:41:13.330470+0000","last_undegraded":"2023-10-06T01:41:13.330470+0000","last_fullsized":"2023-10-06T01:41:13.330470+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:05:22.934401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.337002+0000","last_change":"2023-10-06T01:41:12.341990+0000","last_active":"2023-10-06T01:41:13.337002+0000","last_peered":"2023-10-06T01:41:13.337002+0000","last_clean":"2023-10-06T01:41:13.337002+0000","last_became_active":"2023-10-06T01:41:12.341680+0000","last_became_peered":"2023-10-06T01:41:12.341680+0000","last_unstale":"2023-10-06T01:41:13.337002+0000","last_undegraded":"2023-10-06T01:41:13.337002+0000","last_fullsized":"2023-10-06T01:41:13.337002+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:50:44.552768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.349056+0000","last_change":"2023-10-06T01:41:13.344589+0000","last_active":"2023-10-06T01:41:13.349056+0000","last_peered":"2023-10-06T01:41:13.349056+0000","last_clean":"2023-10-06T01:41:13.349056+0000","last_became_active":"2023-10-06T01:41:13.344281+0000","last_became_peered":"2023-10-06T01:41:13.344281+0000","last_unstale":"2023-10-06T01:41:13.349056+0000","last_undegraded":"2023-10-06T01:41:13.349056+0000","last_fullsized":"2023-10-06T01:41:13.349056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:32:17.938309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"19'1","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.345810+0000","last_change":"2023-10-06T01:41:12.347655+0000","last_active":"2023-10-06T01:41:13.345810+0000","last_peered":"2023-10-06T01:41:13.345810+0000","last_clean":"2023-10-06T01:41:13.345810+0000","last_became_active":"2023-10-06T01:41:12.347353+0000","last_became_peered":"2023-10-06T01:41:12.347353+0000","last_unstale":"2023-10-06T01:41:13.345810+0000","last_undegraded":"2023-10-06T01:41:13.345810+0000","last_fullsized":"2023-10-06T01:41:13.345810+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:49:54.658088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352446+0000","last_change":"2023-10-06T01:41:13.349874+0000","last_active":"2023-10-06T01:41:13.352446+0000","last_peered":"2023-10-06T01:41:13.352446+0000","last_clean":"2023-10-06T01:41:13.352446+0000","last_became_active":"2023-10-06T01:41:13.349607+0000","last_became_peered":"2023-10-06T01:41:13.349607+0000","last_unstale":"2023-10-06T01:41:13.352446+0000","last_undegraded":"2023-10-06T01:41:13.352446+0000","last_fullsized":"2023-10-06T01:41:13.352446+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:06:20.036081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352575+0000","last_change":"2023-10-06T01:41:12.341511+0000","last_active":"2023-10-06T01:41:13.352575+0000","last_peered":"2023-10-06T01:41:13.352575+0000","last_clean":"2023-10-06T01:41:13.352575+0000","last_became_active":"2023-10-06T01:41:12.341216+0000","last_became_peered":"2023-10-06T01:41:12.341216+0000","last_unstale":"2023-10-06T01:41:13.352575+0000","last_undegraded":"2023-10-06T01:41:13.352575+0000","last_fullsized":"2023-10-06T01:41:13.352575+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:27:01.582307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351635+0000","last_change":"2023-10-06T01:41:13.343780+0000","last_active":"2023-10-06T01:41:13.351635+0000","last_peered":"2023-10-06T01:41:13.351635+0000","last_clean":"2023-10-06T01:41:13.351635+0000","last_became_active":"2023-10-06T01:41:13.343515+0000","last_became_peered":"2023-10-06T01:41:13.343515+0000","last_unstale":"2023-10-06T01:41:13.351635+0000","last_undegraded":"2023-10-06T01:41:13.351635+0000","last_fullsized":"2023-10-06T01:41:13.351635+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:38:58.407819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.353702+0000","last_change":"2023-10-06T01:41:12.341780+0000","last_active":"2023-10-06T01:41:13.353702+0000","last_peered":"2023-10-06T01:41:13.353702+0000","last_clean":"2023-10-06T01:41:13.353702+0000","last_became_active":"2023-10-06T01:41:12.340585+0000","last_became_peered":"2023-10-06T01:41:12.340585+0000","last_unstale":"2023-10-06T01:41:13.353702+0000","last_undegraded":"2023-10-06T01:41:13.353702+0000","last_fullsized":"2023-10-06T01:41:13.353702+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:49:54.658088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354558+0000","last_change":"2023-10-06T01:41:13.345543+0000","last_active":"2023-10-06T01:41:13.354558+0000","last_peered":"2023-10-06T01:41:13.354558+0000","last_clean":"2023-10-06T01:41:13.354558+0000","last_became_active":"2023-10-06T01:41:13.345286+0000","last_became_peered":"2023-10-06T01:41:13.345286+0000","last_unstale":"2023-10-06T01:41:13.354558+0000","last_undegraded":"2023-10-06T01:41:13.354558+0000","last_fullsized":"2023-10-06T01:41:13.354558+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:52:48.286318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"19'1","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351829+0000","last_change":"2023-10-06T01:41:12.340776+0000","last_active":"2023-10-06T01:41:13.351829+0000","last_peered":"2023-10-06T01:41:13.351829+0000","last_clean":"2023-10-06T01:41:13.351829+0000","last_became_active":"2023-10-06T01:41:12.340294+0000","last_became_peered":"2023-10-06T01:41:12.340294+0000","last_unstale":"2023-10-06T01:41:13.351829+0000","last_undegraded":"2023-10-06T01:41:13.351829+0000","last_fullsized":"2023-10-06T01:41:13.351829+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:37:13.672716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.357969+0000","last_change":"2023-10-06T01:41:13.355273+0000","last_active":"2023-10-06T01:41:13.357969+0000","last_peered":"2023-10-06T01:41:13.357969+0000","last_clean":"2023-10-06T01:41:13.357969+0000","last_became_active":"2023-10-06T01:41:13.354977+0000","last_became_peered":"2023-10-06T01:41:13.354977+0000","last_unstale":"2023-10-06T01:41:13.357969+0000","last_undegraded":"2023-10-06T01:41:13.357969+0000","last_fullsized":"2023-10-06T01:41:13.357969+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:21:42.851681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"19'4","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.345580+0000","last_change":"2023-10-06T01:41:12.338431+0000","last_active":"2023-10-06T01:41:13.345580+0000","last_peered":"2023-10-06T01:41:13.345580+0000","last_clean":"2023-10-06T01:41:13.345580+0000","last_became_active":"2023-10-06T01:41:12.338081+0000","last_became_peered":"2023-10-06T01:41:12.338081+0000","last_unstale":"2023-10-06T01:41:13.345580+0000","last_undegraded":"2023-10-06T01:41:13.345580+0000","last_fullsized":"2023-10-06T01:41:13.345580+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:16:14.826869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351157+0000","last_change":"2023-10-06T01:41:13.342212+0000","last_active":"2023-10-06T01:41:13.351157+0000","last_peered":"2023-10-06T01:41:13.351157+0000","last_clean":"2023-10-06T01:41:13.351157+0000","last_became_active":"2023-10-06T01:41:13.341948+0000","last_became_peered":"2023-10-06T01:41:13.341948+0000","last_unstale":"2023-10-06T01:41:13.351157+0000","last_undegraded":"2023-10-06T01:41:13.351157+0000","last_fullsized":"2023-10-06T01:41:13.351157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:18:23.688664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354188+0000","last_change":"2023-10-06T01:41:13.348482+0000","last_active":"2023-10-06T01:41:13.354188+0000","last_peered":"2023-10-06T01:41:13.354188+0000","last_clean":"2023-10-06T01:41:13.354188+0000","last_became_active":"2023-10-06T01:41:13.348075+0000","last_became_peered":"2023-10-06T01:41:13.348075+0000","last_unstale":"2023-10-06T01:41:13.354188+0000","last_undegraded":"2023-10-06T01:41:13.354188+0000","last_fullsized":"2023-10-06T01:41:13.354188+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:25:04.563235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.357958+0000","last_change":"2023-10-06T01:41:13.355216+0000","last_active":"2023-10-06T01:41:13.357958+0000","last_peered":"2023-10-06T01:41:13.357958+0000","last_clean":"2023-10-06T01:41:13.357958+0000","last_became_active":"2023-10-06T01:41:13.354853+0000","last_became_peered":"2023-10-06T01:41:13.354853+0000","last_unstale":"2023-10-06T01:41:13.357958+0000","last_undegraded":"2023-10-06T01:41:13.357958+0000","last_fullsized":"2023-10-06T01:41:13.357958+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:42:44.799267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351163+0000","last_change":"2023-10-06T01:41:13.342888+0000","last_active":"2023-10-06T01:41:13.351163+0000","last_peered":"2023-10-06T01:41:13.351163+0000","last_clean":"2023-10-06T01:41:13.351163+0000","last_became_active":"2023-10-06T01:41:13.342626+0000","last_became_peered":"2023-10-06T01:41:13.342626+0000","last_unstale":"2023-10-06T01:41:13.351163+0000","last_undegraded":"2023-10-06T01:41:13.351163+0000","last_fullsized":"2023-10-06T01:41:13.351163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:43:29.491858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.357705+0000","last_change":"2023-10-06T01:41:13.353822+0000","last_active":"2023-10-06T01:41:13.357705+0000","last_peered":"2023-10-06T01:41:13.357705+0000","last_clean":"2023-10-06T01:41:13.357705+0000","last_became_active":"2023-10-06T01:41:13.353292+0000","last_became_peered":"2023-10-06T01:41:13.353292+0000","last_unstale":"2023-10-06T01:41:13.357705+0000","last_undegraded":"2023-10-06T01:41:13.357705+0000","last_fullsized":"2023-10-06T01:41:13.357705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:52:48.286318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356636+0000","last_change":"2023-10-06T01:41:13.353662+0000","last_active":"2023-10-06T01:41:13.356636+0000","last_peered":"2023-10-06T01:41:13.356636+0000","last_clean":"2023-10-06T01:41:13.356636+0000","last_became_active":"2023-10-06T01:41:13.353269+0000","last_became_peered":"2023-10-06T01:41:13.353269+0000","last_unstale":"2023-10-06T01:41:13.356636+0000","last_undegraded":"2023-10-06T01:41:13.356636+0000","last_fullsized":"2023-10-06T01:41:13.356636+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:11:44.715827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298096,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073744,"statfs":{"total":96636764160,"available":96331513856,"internally_reserved":0,"allocated":827392,"data_stored":666304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297484,"kb_used_data":196,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074356,"statfs":{"total":96636764160,"available":96332140544,"internally_reserved":0,"allocated":200704,"data_stored":72831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298068,"kb_used_data":780,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073772,"statfs":{"total":96636764160,"available":96331542528,"internally_reserved":0,"allocated":798720,"data_stored":664552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297488,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074352,"statfs":{"total":96636764160,"available":96332136448,"internally_reserved":0,"allocated":204800,"data_stored":73385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297488,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074352,"statfs":{"total":96636764160,"available":96332136448,"internally_reserved":0,"allocated":204800,"data_stored":74184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297488,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074352,"statfs":{"total":96636764160,"available":96332136448,"internally_reserved":0,"allocated":204800,"data_stored":72921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297508,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074332,"statfs":{"total":96636764160,"available":96332115968,"internally_reserved":0,"allocated":225280,"data_stored":75776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97899999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297492,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074348,"statfs":{"total":96636764160,"available":96332132352,"internally_reserved":0,"allocated":208896,"data_stored":73455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.372}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-06T01:41:26.197 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-06T01:41:26.349 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.347+0000 7f1c1676c700 1 Processor -- start 2023-10-06T01:41:26.349 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.348+0000 7f1c1676c700 1 -- start start 2023-10-06T01:41:26.351 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.348+0000 7f1c1676c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1c10151a60 0x7f1c100b0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:26.351 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.348+0000 7f1c1676c700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f1c10158840 0x7f1c100b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:26.352 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.348+0000 7f1c1676c700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1c100aefe0 0x7f1c100af460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:26.352 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.348+0000 7f1c1676c700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c100af9a0 con 0x7f1c10151a60 2023-10-06T01:41:26.352 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.348+0000 7f1c1676c700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c100cb770 con 0x7f1c10158840 2023-10-06T01:41:26.352 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.348+0000 7f1c1676c700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c100cba50 con 0x7f1c100aefe0 2023-10-06T01:41:26.352 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.349+0000 7f1c0ffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1c10151a60 0x7f1c100b0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:26.352 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.349+0000 7f1c0ffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1c10151a60 0x7f1c100b0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52260/0 (socket says 172.21.15.27:52260) 2023-10-06T01:41:26.352 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.349+0000 7f1c0ffff700 1 -- 172.21.15.27:0/1180723689 learned_addr learned my addr 172.21.15.27:0/1180723689 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:26.352 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.349+0000 7f1c14d09700 1 --2- 172.21.15.27:0/1180723689 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1c100aefe0 0x7f1c100af460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:26.352 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.349+0000 7f1c07fff700 1 --2- 172.21.15.27:0/1180723689 >> v2:172.21.15.124:3300/0 conn(0x7f1c10158840 0x7f1c100b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:26.352 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.349+0000 7f1c0ffff700 1 -- 172.21.15.27:0/1180723689 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1c100aefe0 msgr2=0x7f1c100af460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:26.353 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.349+0000 7f1c0ffff700 1 --2- 172.21.15.27:0/1180723689 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1c100aefe0 0x7f1c100af460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:26.353 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.349+0000 7f1c0ffff700 1 -- 172.21.15.27:0/1180723689 >> v2:172.21.15.124:3300/0 conn(0x7f1c10158840 msgr2=0x7f1c100b29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:26.353 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.349+0000 7f1c0ffff700 1 --2- 172.21.15.27:0/1180723689 >> v2:172.21.15.124:3300/0 conn(0x7f1c10158840 0x7f1c100b29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:26.353 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.349+0000 7f1c0ffff700 1 -- 172.21.15.27:0/1180723689 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c100cbd30 con 0x7f1c10151a60 2023-10-06T01:41:26.353 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.349+0000 7f1c0ffff700 1 --2- 172.21.15.27:0/1180723689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c10151a60 0x7f1c100b0880 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1c10067930 tx=0x7f1c00009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f822effc9aeb6526 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:26.353 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.349+0000 7f1c0f7fe700 1 -- 172.21.15.27:0/1180723689 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0000b860 con 0x7f1c10151a60 2023-10-06T01:41:26.353 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.349+0000 7f1c0f7fe700 1 -- 172.21.15.27:0/1180723689 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c0000b9c0 con 0x7f1c10151a60 2023-10-06T01:41:26.353 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.349+0000 7f1c0f7fe700 1 -- 172.21.15.27:0/1180723689 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0000bd60 con 0x7f1c10151a60 2023-10-06T01:41:26.353 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.350+0000 7f1c1676c700 1 -- 172.21.15.27:0/1180723689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c10151a60 msgr2=0x7f1c100b0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:26.353 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.350+0000 7f1c1676c700 1 --2- 172.21.15.27:0/1180723689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c10151a60 0x7f1c100b0880 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1c10067930 tx=0x7f1c00009250 comp rx=0 tx=0).stop 2023-10-06T01:41:26.353 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.350+0000 7f1c1676c700 1 -- 172.21.15.27:0/1180723689 shutdown_connections 2023-10-06T01:41:26.353 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.350+0000 7f1c1676c700 1 --2- 172.21.15.27:0/1180723689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c10151a60 0x7f1c100b0880 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:26.354 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.350+0000 7f1c1676c700 1 --2- 172.21.15.27:0/1180723689 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1c100aefe0 0x7f1c100af460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:26.354 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.350+0000 7f1c1676c700 1 --2- 172.21.15.27:0/1180723689 >> v2:172.21.15.124:3300/0 conn(0x7f1c10158840 0x7f1c100b29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:26.354 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.350+0000 7f1c1676c700 1 -- 172.21.15.27:0/1180723689 >> 172.21.15.27:0/1180723689 conn(0x7f1c1006e6f0 msgr2=0x7f1c1006afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:26.354 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.350+0000 7f1c1676c700 1 -- 172.21.15.27:0/1180723689 shutdown_connections 2023-10-06T01:41:26.354 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.350+0000 7f1c1676c700 1 -- 172.21.15.27:0/1180723689 wait complete. 2023-10-06T01:41:26.357 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.351+0000 7f1c1676c700 1 Processor -- start 2023-10-06T01:41:26.357 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.351+0000 7f1c1676c700 1 -- start start 2023-10-06T01:41:26.357 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.352+0000 7f1c1676c700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1c100aefe0 0x7f1c1016acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:26.357 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.352+0000 7f1c1676c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c10151a60 0x7f1c101676f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:26.357 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.352+0000 7f1c1676c700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1c10158840 0x7f1c100b1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:26.357 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.352+0000 7f1c1676c700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1016bc00 con 0x7f1c10151a60 2023-10-06T01:41:26.357 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.352+0000 7f1c1676c700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1015b2d0 con 0x7f1c100aefe0 2023-10-06T01:41:26.357 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.352+0000 7f1c1676c700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1006f460 con 0x7f1c10158840 2023-10-06T01:41:26.357 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.352+0000 7f1c07fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c10151a60 0x7f1c101676f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:26.357 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.352+0000 7f1c07fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c10151a60 0x7f1c101676f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52276/0 (socket says 172.21.15.27:52276) 2023-10-06T01:41:26.358 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.352+0000 7f1c07fff700 1 -- 172.21.15.27:0/297415040 learned_addr learned my addr 172.21.15.27:0/297415040 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:26.358 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.352+0000 7f1c0ffff700 1 --2- 172.21.15.27:0/297415040 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1c100aefe0 0x7f1c1016acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:26.358 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.352+0000 7f1c14d09700 1 --2- 172.21.15.27:0/297415040 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1c10158840 0x7f1c100b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:26.358 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.352+0000 7f1c07fff700 1 -- 172.21.15.27:0/297415040 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1c10158840 msgr2=0x7f1c100b1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:26.358 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.352+0000 7f1c07fff700 1 --2- 172.21.15.27:0/297415040 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1c10158840 0x7f1c100b1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:26.358 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.352+0000 7f1c07fff700 1 -- 172.21.15.27:0/297415040 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1c100aefe0 msgr2=0x7f1c1016acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:26.358 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.352+0000 7f1c07fff700 1 --2- 172.21.15.27:0/297415040 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1c100aefe0 0x7f1c1016acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:26.358 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.352+0000 7f1c07fff700 1 -- 172.21.15.27:0/297415040 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c000094f0 con 0x7f1c10151a60 2023-10-06T01:41:26.358 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.352+0000 7f1c07fff700 1 --2- 172.21.15.27:0/297415040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c10151a60 0x7f1c101676f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1bf800d890 tx=0x7f1bf8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:26.358 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.352+0000 7f1c0cff9700 1 -- 172.21.15.27:0/297415040 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf80125f0 con 0x7f1c10151a60 2023-10-06T01:41:26.358 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.352+0000 7f1c0cff9700 1 -- 172.21.15.27:0/297415040 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bf8012750 con 0x7f1c10151a60 2023-10-06T01:41:26.359 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.353+0000 7f1c1676c700 1 -- 172.21.15.27:0/297415040 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c1016b360 con 0x7f1c10151a60 2023-10-06T01:41:26.359 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.353+0000 7f1c0cff9700 1 -- 172.21.15.27:0/297415040 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf8012b10 con 0x7f1c10151a60 2023-10-06T01:41:26.359 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.353+0000 7f1c1676c700 1 -- 172.21.15.27:0/297415040 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c1016b920 con 0x7f1c10151a60 2023-10-06T01:41:26.359 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.354+0000 7f1c1676c700 1 -- 172.21.15.27:0/297415040 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c1015b460 con 0x7f1c10151a60 2023-10-06T01:41:26.359 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.354+0000 7f1c0cff9700 1 -- 172.21.15.27:0/297415040 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1bf801e460 con 0x7f1c10151a60 2023-10-06T01:41:26.359 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.354+0000 7f1c0cff9700 1 --2- 172.21.15.27:0/297415040 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1bf0066f30 0x7f1bf00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:26.359 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.355+0000 7f1c0cff9700 1 -- 172.21.15.27:0/297415040 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f1bf8084ba0 con 0x7f1c10151a60 2023-10-06T01:41:26.359 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.355+0000 7f1c0ffff700 1 --2- 172.21.15.27:0/297415040 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1bf0066f30 0x7f1bf00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:26.359 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.355+0000 7f1c0ffff700 1 --2- 172.21.15.27:0/297415040 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1bf0066f30 0x7f1bf00693e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1c10067930 tx=0x7f1c00017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:26.360 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.359+0000 7f1c0cff9700 1 -- 172.21.15.27:0/297415040 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1bf8051c10 con 0x7f1c10151a60 2023-10-06T01:41:26.550 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.548+0000 7f1c1676c700 1 -- 172.21.15.27:0/297415040 --> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1c10064050 con 0x7f1bf0066f30 2023-10-06T01:41:26.558 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.557+0000 7f1c0cff9700 1 -- 172.21.15.27:0/297415040 <== mgr.4100 v2:172.21.15.124:6832/90278 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286802 (secure 0 0 0) 0x7f1c10064050 con 0x7f1bf0066f30 2023-10-06T01:41:26.561 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:26.564 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.560+0000 7f1c1676c700 1 -- 172.21.15.27:0/297415040 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1bf0066f30 msgr2=0x7f1bf00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:26.564 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.560+0000 7f1c1676c700 1 --2- 172.21.15.27:0/297415040 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1bf0066f30 0x7f1bf00693e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1c10067930 tx=0x7f1c00017000 comp rx=0 tx=0).stop 2023-10-06T01:41:26.564 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.560+0000 7f1c1676c700 1 -- 172.21.15.27:0/297415040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c10151a60 msgr2=0x7f1c101676f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:26.564 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.560+0000 7f1c1676c700 1 --2- 172.21.15.27:0/297415040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c10151a60 0x7f1c101676f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f1bf800d890 tx=0x7f1bf8008040 comp rx=0 tx=0).stop 2023-10-06T01:41:26.564 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.560+0000 7f1c1676c700 1 -- 172.21.15.27:0/297415040 shutdown_connections 2023-10-06T01:41:26.564 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.560+0000 7f1c1676c700 1 --2- 172.21.15.27:0/297415040 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1bf0066f30 0x7f1bf00693e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:26.564 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.560+0000 7f1c1676c700 1 --2- 172.21.15.27:0/297415040 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1c10158840 0x7f1c100b1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:26.564 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.560+0000 7f1c1676c700 1 --2- 172.21.15.27:0/297415040 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1c100aefe0 0x7f1c1016acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:26.564 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.560+0000 7f1c1676c700 1 --2- 172.21.15.27:0/297415040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1c10151a60 0x7f1c101676f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:26.564 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.560+0000 7f1c1676c700 1 -- 172.21.15.27:0/297415040 >> 172.21.15.27:0/297415040 conn(0x7f1c1006e6f0 msgr2=0x7f1c100b0880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:26.564 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.560+0000 7f1c1676c700 1 -- 172.21.15.27:0/297415040 shutdown_connections 2023-10-06T01:41:26.565 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.560+0000 7f1c1676c700 1 -- 172.21.15.27:0/297415040 wait complete. 2023-10-06T01:41:26.565 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-06T01:41:26.574 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2023-10-06T01:41:26.041299+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":210,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2381112,"kb_used_data":2808,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752593608,"statfs":{"total":773094113280,"available":770655854592,"internally_reserved":0,"allocated":2875392,"data_stored":1773408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.713509"},"pg_stats":[{"pgid":"4.3f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.349688+0000","last_change":"2023-10-06T01:41:13.346440+0000","last_active":"2023-10-06T01:41:13.349688+0000","last_peered":"2023-10-06T01:41:13.349688+0000","last_clean":"2023-10-06T01:41:13.349688+0000","last_became_active":"2023-10-06T01:41:13.343345+0000","last_became_peered":"2023-10-06T01:41:13.343345+0000","last_unstale":"2023-10-06T01:41:13.349688+0000","last_undegraded":"2023-10-06T01:41:13.349688+0000","last_fullsized":"2023-10-06T01:41:13.349688+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:15:41.848935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354765+0000","last_change":"2023-10-06T01:41:13.348847+0000","last_active":"2023-10-06T01:41:13.354765+0000","last_peered":"2023-10-06T01:41:13.354765+0000","last_clean":"2023-10-06T01:41:13.354765+0000","last_became_active":"2023-10-06T01:41:13.348585+0000","last_became_peered":"2023-10-06T01:41:13.348585+0000","last_unstale":"2023-10-06T01:41:13.354765+0000","last_undegraded":"2023-10-06T01:41:13.354765+0000","last_fullsized":"2023-10-06T01:41:13.354765+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:31:20.840764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354463+0000","last_change":"2023-10-06T01:41:13.345122+0000","last_active":"2023-10-06T01:41:13.354463+0000","last_peered":"2023-10-06T01:41:13.354463+0000","last_clean":"2023-10-06T01:41:13.354463+0000","last_became_active":"2023-10-06T01:41:13.344752+0000","last_became_peered":"2023-10-06T01:41:13.344752+0000","last_unstale":"2023-10-06T01:41:13.354463+0000","last_undegraded":"2023-10-06T01:41:13.354463+0000","last_fullsized":"2023-10-06T01:41:13.354463+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:42:56.041827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.342734+0000","last_change":"2023-10-06T01:41:13.340589+0000","last_active":"2023-10-06T01:41:13.342734+0000","last_peered":"2023-10-06T01:41:13.342734+0000","last_clean":"2023-10-06T01:41:13.342734+0000","last_became_active":"2023-10-06T01:41:13.340339+0000","last_became_peered":"2023-10-06T01:41:13.340339+0000","last_unstale":"2023-10-06T01:41:13.342734+0000","last_undegraded":"2023-10-06T01:41:13.342734+0000","last_fullsized":"2023-10-06T01:41:13.342734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:32:17.938309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.346061+0000","last_change":"2023-10-06T01:41:13.342002+0000","last_active":"2023-10-06T01:41:13.346061+0000","last_peered":"2023-10-06T01:41:13.346061+0000","last_clean":"2023-10-06T01:41:13.346061+0000","last_became_active":"2023-10-06T01:41:13.341403+0000","last_became_peered":"2023-10-06T01:41:13.341403+0000","last_unstale":"2023-10-06T01:41:13.346061+0000","last_undegraded":"2023-10-06T01:41:13.346061+0000","last_fullsized":"2023-10-06T01:41:13.346061+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:34:55.375989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354785+0000","last_change":"2023-10-06T01:41:13.345138+0000","last_active":"2023-10-06T01:41:13.354785+0000","last_peered":"2023-10-06T01:41:13.354785+0000","last_clean":"2023-10-06T01:41:13.354785+0000","last_became_active":"2023-10-06T01:41:13.344810+0000","last_became_peered":"2023-10-06T01:41:13.344810+0000","last_unstale":"2023-10-06T01:41:13.354785+0000","last_undegraded":"2023-10-06T01:41:13.354785+0000","last_fullsized":"2023-10-06T01:41:13.354785+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:17:59.027125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.39","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356675+0000","last_change":"2023-10-06T01:41:13.352141+0000","last_active":"2023-10-06T01:41:13.356675+0000","last_peered":"2023-10-06T01:41:13.356675+0000","last_clean":"2023-10-06T01:41:13.356675+0000","last_became_active":"2023-10-06T01:41:13.351628+0000","last_became_peered":"2023-10-06T01:41:13.351628+0000","last_unstale":"2023-10-06T01:41:13.356675+0000","last_undegraded":"2023-10-06T01:41:13.356675+0000","last_fullsized":"2023-10-06T01:41:13.356675+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:03:37.385158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.38","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354278+0000","last_change":"2023-10-06T01:41:13.345960+0000","last_active":"2023-10-06T01:41:13.354278+0000","last_peered":"2023-10-06T01:41:13.354278+0000","last_clean":"2023-10-06T01:41:13.354278+0000","last_became_active":"2023-10-06T01:41:13.345702+0000","last_became_peered":"2023-10-06T01:41:13.345702+0000","last_unstale":"2023-10-06T01:41:13.354278+0000","last_undegraded":"2023-10-06T01:41:13.354278+0000","last_fullsized":"2023-10-06T01:41:13.354278+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:03:37.385158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.37","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.348168+0000","last_change":"2023-10-06T01:41:13.343792+0000","last_active":"2023-10-06T01:41:13.348168+0000","last_peered":"2023-10-06T01:41:13.348168+0000","last_clean":"2023-10-06T01:41:13.348168+0000","last_became_active":"2023-10-06T01:41:13.343432+0000","last_became_peered":"2023-10-06T01:41:13.343432+0000","last_unstale":"2023-10-06T01:41:13.348168+0000","last_undegraded":"2023-10-06T01:41:13.348168+0000","last_fullsized":"2023-10-06T01:41:13.348168+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:52:48.286318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.36","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354747+0000","last_change":"2023-10-06T01:41:13.347399+0000","last_active":"2023-10-06T01:41:13.354747+0000","last_peered":"2023-10-06T01:41:13.354747+0000","last_clean":"2023-10-06T01:41:13.354747+0000","last_became_active":"2023-10-06T01:41:13.347136+0000","last_became_peered":"2023-10-06T01:41:13.347136+0000","last_unstale":"2023-10-06T01:41:13.354747+0000","last_undegraded":"2023-10-06T01:41:13.354747+0000","last_fullsized":"2023-10-06T01:41:13.354747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:42:23.199959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.35","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354772+0000","last_change":"2023-10-06T01:41:13.350059+0000","last_active":"2023-10-06T01:41:13.354772+0000","last_peered":"2023-10-06T01:41:13.354772+0000","last_clean":"2023-10-06T01:41:13.354772+0000","last_became_active":"2023-10-06T01:41:13.349673+0000","last_became_peered":"2023-10-06T01:41:13.349673+0000","last_unstale":"2023-10-06T01:41:13.354772+0000","last_undegraded":"2023-10-06T01:41:13.354772+0000","last_fullsized":"2023-10-06T01:41:13.354772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:12:57.315612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.34","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351538+0000","last_change":"2023-10-06T01:41:13.348001+0000","last_active":"2023-10-06T01:41:13.351538+0000","last_peered":"2023-10-06T01:41:13.351538+0000","last_clean":"2023-10-06T01:41:13.351538+0000","last_became_active":"2023-10-06T01:41:13.347719+0000","last_became_peered":"2023-10-06T01:41:13.347719+0000","last_unstale":"2023-10-06T01:41:13.351538+0000","last_undegraded":"2023-10-06T01:41:13.351538+0000","last_fullsized":"2023-10-06T01:41:13.351538+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:37:53.150671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.33","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.355181+0000","last_change":"2023-10-06T01:41:13.352875+0000","last_active":"2023-10-06T01:41:13.355181+0000","last_peered":"2023-10-06T01:41:13.355181+0000","last_clean":"2023-10-06T01:41:13.355181+0000","last_became_active":"2023-10-06T01:41:13.352589+0000","last_became_peered":"2023-10-06T01:41:13.352589+0000","last_unstale":"2023-10-06T01:41:13.355181+0000","last_undegraded":"2023-10-06T01:41:13.355181+0000","last_fullsized":"2023-10-06T01:41:13.355181+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:49:55.665032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.32","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354387+0000","last_change":"2023-10-06T01:41:13.348362+0000","last_active":"2023-10-06T01:41:13.354387+0000","last_peered":"2023-10-06T01:41:13.354387+0000","last_clean":"2023-10-06T01:41:13.354387+0000","last_became_active":"2023-10-06T01:41:13.348181+0000","last_became_peered":"2023-10-06T01:41:13.348181+0000","last_unstale":"2023-10-06T01:41:13.354387+0000","last_undegraded":"2023-10-06T01:41:13.354387+0000","last_fullsized":"2023-10-06T01:41:13.354387+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:27:54.074728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.31","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356697+0000","last_change":"2023-10-06T01:41:13.352048+0000","last_active":"2023-10-06T01:41:13.356697+0000","last_peered":"2023-10-06T01:41:13.356697+0000","last_clean":"2023-10-06T01:41:13.356697+0000","last_became_active":"2023-10-06T01:41:13.351640+0000","last_became_peered":"2023-10-06T01:41:13.351640+0000","last_unstale":"2023-10-06T01:41:13.356697+0000","last_undegraded":"2023-10-06T01:41:13.356697+0000","last_fullsized":"2023-10-06T01:41:13.356697+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:42:23.199959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.30","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354602+0000","last_change":"2023-10-06T01:41:13.345677+0000","last_active":"2023-10-06T01:41:13.354602+0000","last_peered":"2023-10-06T01:41:13.354602+0000","last_clean":"2023-10-06T01:41:13.354602+0000","last_became_active":"2023-10-06T01:41:13.345257+0000","last_became_peered":"2023-10-06T01:41:13.345257+0000","last_unstale":"2023-10-06T01:41:13.354602+0000","last_undegraded":"2023-10-06T01:41:13.354602+0000","last_fullsized":"2023-10-06T01:41:13.354602+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:27:07.040029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.348233+0000","last_change":"2023-10-06T01:41:13.344490+0000","last_active":"2023-10-06T01:41:13.348233+0000","last_peered":"2023-10-06T01:41:13.348233+0000","last_clean":"2023-10-06T01:41:13.348233+0000","last_became_active":"2023-10-06T01:41:13.344220+0000","last_became_peered":"2023-10-06T01:41:13.344220+0000","last_unstale":"2023-10-06T01:41:13.348233+0000","last_undegraded":"2023-10-06T01:41:13.348233+0000","last_fullsized":"2023-10-06T01:41:13.348233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T01:51:57.229301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354756+0000","last_change":"2023-10-06T01:41:13.350575+0000","last_active":"2023-10-06T01:41:13.354756+0000","last_peered":"2023-10-06T01:41:13.354756+0000","last_clean":"2023-10-06T01:41:13.354756+0000","last_became_active":"2023-10-06T01:41:13.350311+0000","last_became_peered":"2023-10-06T01:41:13.350311+0000","last_unstale":"2023-10-06T01:41:13.354756+0000","last_undegraded":"2023-10-06T01:41:13.354756+0000","last_fullsized":"2023-10-06T01:41:13.354756+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:39:03.091509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351594+0000","last_change":"2023-10-06T01:41:13.347386+0000","last_active":"2023-10-06T01:41:13.351594+0000","last_peered":"2023-10-06T01:41:13.351594+0000","last_clean":"2023-10-06T01:41:13.351594+0000","last_became_active":"2023-10-06T01:41:13.347101+0000","last_became_peered":"2023-10-06T01:41:13.347101+0000","last_unstale":"2023-10-06T01:41:13.351594+0000","last_undegraded":"2023-10-06T01:41:13.351594+0000","last_fullsized":"2023-10-06T01:41:13.351594+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:13:15.011521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351504+0000","last_change":"2023-10-06T01:41:13.342640+0000","last_active":"2023-10-06T01:41:13.351504+0000","last_peered":"2023-10-06T01:41:13.351504+0000","last_clean":"2023-10-06T01:41:13.351504+0000","last_became_active":"2023-10-06T01:41:13.342157+0000","last_became_peered":"2023-10-06T01:41:13.342157+0000","last_unstale":"2023-10-06T01:41:13.351504+0000","last_undegraded":"2023-10-06T01:41:13.351504+0000","last_fullsized":"2023-10-06T01:41:13.351504+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:07:52.521701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356729+0000","last_change":"2023-10-06T01:41:13.352582+0000","last_active":"2023-10-06T01:41:13.356729+0000","last_peered":"2023-10-06T01:41:13.356729+0000","last_clean":"2023-10-06T01:41:13.356729+0000","last_became_active":"2023-10-06T01:41:13.352281+0000","last_became_peered":"2023-10-06T01:41:13.352281+0000","last_unstale":"2023-10-06T01:41:13.356729+0000","last_undegraded":"2023-10-06T01:41:13.356729+0000","last_fullsized":"2023-10-06T01:41:13.356729+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:31:20.840764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351592+0000","last_change":"2023-10-06T01:41:13.345324+0000","last_active":"2023-10-06T01:41:13.351592+0000","last_peered":"2023-10-06T01:41:13.351592+0000","last_clean":"2023-10-06T01:41:13.351592+0000","last_became_active":"2023-10-06T01:41:13.344997+0000","last_became_peered":"2023-10-06T01:41:13.344997+0000","last_unstale":"2023-10-06T01:41:13.351592+0000","last_undegraded":"2023-10-06T01:41:13.351592+0000","last_fullsized":"2023-10-06T01:41:13.351592+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:27:54.074728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.29","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.357943+0000","last_change":"2023-10-06T01:41:13.355108+0000","last_active":"2023-10-06T01:41:13.357943+0000","last_peered":"2023-10-06T01:41:13.357943+0000","last_clean":"2023-10-06T01:41:13.357943+0000","last_became_active":"2023-10-06T01:41:13.354621+0000","last_became_peered":"2023-10-06T01:41:13.354621+0000","last_unstale":"2023-10-06T01:41:13.357943+0000","last_undegraded":"2023-10-06T01:41:13.357943+0000","last_fullsized":"2023-10-06T01:41:13.357943+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:34:55.375989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.28","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356702+0000","last_change":"2023-10-06T01:41:13.352301+0000","last_active":"2023-10-06T01:41:13.356702+0000","last_peered":"2023-10-06T01:41:13.356702+0000","last_clean":"2023-10-06T01:41:13.356702+0000","last_became_active":"2023-10-06T01:41:13.351858+0000","last_became_peered":"2023-10-06T01:41:13.351858+0000","last_unstale":"2023-10-06T01:41:13.356702+0000","last_undegraded":"2023-10-06T01:41:13.356702+0000","last_fullsized":"2023-10-06T01:41:13.356702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:42:46.214805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.27","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.348956+0000","last_change":"2023-10-06T01:41:13.344094+0000","last_active":"2023-10-06T01:41:13.348956+0000","last_peered":"2023-10-06T01:41:13.348956+0000","last_clean":"2023-10-06T01:41:13.348956+0000","last_became_active":"2023-10-06T01:41:13.343796+0000","last_became_peered":"2023-10-06T01:41:13.343796+0000","last_unstale":"2023-10-06T01:41:13.348956+0000","last_undegraded":"2023-10-06T01:41:13.348956+0000","last_fullsized":"2023-10-06T01:41:13.348956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:21:30.451862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.26","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.355700+0000","last_change":"2023-10-06T01:41:13.350352+0000","last_active":"2023-10-06T01:41:13.355700+0000","last_peered":"2023-10-06T01:41:13.355700+0000","last_clean":"2023-10-06T01:41:13.355700+0000","last_became_active":"2023-10-06T01:41:13.350133+0000","last_became_peered":"2023-10-06T01:41:13.350133+0000","last_unstale":"2023-10-06T01:41:13.355700+0000","last_undegraded":"2023-10-06T01:41:13.355700+0000","last_fullsized":"2023-10-06T01:41:13.355700+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:05:37.961571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.25","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354695+0000","last_change":"2023-10-06T01:41:13.353241+0000","last_active":"2023-10-06T01:41:13.354695+0000","last_peered":"2023-10-06T01:41:13.354695+0000","last_clean":"2023-10-06T01:41:13.354695+0000","last_became_active":"2023-10-06T01:41:13.352824+0000","last_became_peered":"2023-10-06T01:41:13.352824+0000","last_unstale":"2023-10-06T01:41:13.354695+0000","last_undegraded":"2023-10-06T01:41:13.354695+0000","last_fullsized":"2023-10-06T01:41:13.354695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:52:48.286318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"19'1","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.348819+0000","last_change":"2023-10-06T01:41:12.339675+0000","last_active":"2023-10-06T01:41:13.348819+0000","last_peered":"2023-10-06T01:41:13.348819+0000","last_clean":"2023-10-06T01:41:13.348819+0000","last_became_active":"2023-10-06T01:41:12.339258+0000","last_became_peered":"2023-10-06T01:41:12.339258+0000","last_unstale":"2023-10-06T01:41:13.348819+0000","last_undegraded":"2023-10-06T01:41:13.348819+0000","last_fullsized":"2023-10-06T01:41:13.348819+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:49:54.658088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.357395+0000","last_change":"2023-10-06T01:41:13.352444+0000","last_active":"2023-10-06T01:41:13.357395+0000","last_peered":"2023-10-06T01:41:13.357395+0000","last_clean":"2023-10-06T01:41:13.357395+0000","last_became_active":"2023-10-06T01:41:13.352134+0000","last_became_peered":"2023-10-06T01:41:13.352134+0000","last_unstale":"2023-10-06T01:41:13.357395+0000","last_undegraded":"2023-10-06T01:41:13.357395+0000","last_fullsized":"2023-10-06T01:41:13.357395+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:11:26.794025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.336930+0000","last_change":"2023-10-06T01:41:12.347654+0000","last_active":"2023-10-06T01:41:13.336930+0000","last_peered":"2023-10-06T01:41:13.336930+0000","last_clean":"2023-10-06T01:41:13.336930+0000","last_became_active":"2023-10-06T01:41:12.347386+0000","last_became_peered":"2023-10-06T01:41:12.347386+0000","last_unstale":"2023-10-06T01:41:13.336930+0000","last_undegraded":"2023-10-06T01:41:13.336930+0000","last_fullsized":"2023-10-06T01:41:13.336930+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:07:30.529679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356475+0000","last_change":"2023-10-06T01:41:13.352350+0000","last_active":"2023-10-06T01:41:13.356475+0000","last_peered":"2023-10-06T01:41:13.356475+0000","last_clean":"2023-10-06T01:41:13.356475+0000","last_became_active":"2023-10-06T01:41:13.352091+0000","last_became_peered":"2023-10-06T01:41:13.352091+0000","last_unstale":"2023-10-06T01:41:13.356475+0000","last_undegraded":"2023-10-06T01:41:13.356475+0000","last_fullsized":"2023-10-06T01:41:13.356475+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:31:20.840764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.334517+0000","last_change":"2023-10-06T01:41:12.343086+0000","last_active":"2023-10-06T01:41:13.334517+0000","last_peered":"2023-10-06T01:41:13.334517+0000","last_clean":"2023-10-06T01:41:13.334517+0000","last_became_active":"2023-10-06T01:41:12.342595+0000","last_became_peered":"2023-10-06T01:41:12.342595+0000","last_unstale":"2023-10-06T01:41:13.334517+0000","last_undegraded":"2023-10-06T01:41:13.334517+0000","last_fullsized":"2023-10-06T01:41:13.334517+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:15:40.841990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351933+0000","last_change":"2023-10-06T01:41:13.347027+0000","last_active":"2023-10-06T01:41:13.351933+0000","last_peered":"2023-10-06T01:41:13.351933+0000","last_clean":"2023-10-06T01:41:13.351933+0000","last_became_active":"2023-10-06T01:41:13.346469+0000","last_became_peered":"2023-10-06T01:41:13.346469+0000","last_unstale":"2023-10-06T01:41:13.351933+0000","last_undegraded":"2023-10-06T01:41:13.351933+0000","last_fullsized":"2023-10-06T01:41:13.351933+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:37:14.679660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.329704+0000","last_change":"2023-10-06T01:41:12.341922+0000","last_active":"2023-10-06T01:41:13.329704+0000","last_peered":"2023-10-06T01:41:13.329704+0000","last_clean":"2023-10-06T01:41:13.329704+0000","last_became_active":"2023-10-06T01:41:12.341543+0000","last_became_peered":"2023-10-06T01:41:12.341543+0000","last_unstale":"2023-10-06T01:41:13.329704+0000","last_undegraded":"2023-10-06T01:41:13.329704+0000","last_fullsized":"2023-10-06T01:41:13.329704+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:13:14.004577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352198+0000","last_change":"2023-10-06T01:41:13.345274+0000","last_active":"2023-10-06T01:41:13.352198+0000","last_peered":"2023-10-06T01:41:13.352198+0000","last_clean":"2023-10-06T01:41:13.352198+0000","last_became_active":"2023-10-06T01:41:13.344992+0000","last_became_peered":"2023-10-06T01:41:13.344992+0000","last_unstale":"2023-10-06T01:41:13.352198+0000","last_undegraded":"2023-10-06T01:41:13.352198+0000","last_fullsized":"2023-10-06T01:41:13.352198+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:39:03.091509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.339176+0000","last_change":"2023-10-06T01:41:12.343531+0000","last_active":"2023-10-06T01:41:13.339176+0000","last_peered":"2023-10-06T01:41:13.339176+0000","last_clean":"2023-10-06T01:41:13.339176+0000","last_became_active":"2023-10-06T01:41:12.343256+0000","last_became_peered":"2023-10-06T01:41:12.343256+0000","last_unstale":"2023-10-06T01:41:13.339176+0000","last_undegraded":"2023-10-06T01:41:13.339176+0000","last_fullsized":"2023-10-06T01:41:13.339176+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:16:14.826869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.358071+0000","last_change":"2023-10-06T01:41:13.357361+0000","last_active":"2023-10-06T01:41:13.358071+0000","last_peered":"2023-10-06T01:41:13.358071+0000","last_clean":"2023-10-06T01:41:13.358071+0000","last_became_active":"2023-10-06T01:41:13.356980+0000","last_became_peered":"2023-10-06T01:41:13.356980+0000","last_unstale":"2023-10-06T01:41:13.358071+0000","last_undegraded":"2023-10-06T01:41:13.358071+0000","last_fullsized":"2023-10-06T01:41:13.358071+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:42:51.581939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"19'3","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.355632+0000","last_change":"2023-10-06T01:41:12.350010+0000","last_active":"2023-10-06T01:41:13.355632+0000","last_peered":"2023-10-06T01:41:13.355632+0000","last_clean":"2023-10-06T01:41:13.355632+0000","last_became_active":"2023-10-06T01:41:12.349758+0000","last_became_peered":"2023-10-06T01:41:12.349758+0000","last_unstale":"2023-10-06T01:41:13.355632+0000","last_undegraded":"2023-10-06T01:41:13.355632+0000","last_fullsized":"2023-10-06T01:41:13.355632+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:12:51.734395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352993+0000","last_change":"2023-10-06T01:41:13.348953+0000","last_active":"2023-10-06T01:41:13.352993+0000","last_peered":"2023-10-06T01:41:13.352993+0000","last_clean":"2023-10-06T01:41:13.352993+0000","last_became_active":"2023-10-06T01:41:13.348683+0000","last_became_peered":"2023-10-06T01:41:13.348683+0000","last_unstale":"2023-10-06T01:41:13.352993+0000","last_undegraded":"2023-10-06T01:41:13.352993+0000","last_fullsized":"2023-10-06T01:41:13.352993+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:42:46.214805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"19'6","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.357074+0000","last_change":"2023-10-06T01:41:12.340922+0000","last_active":"2023-10-06T01:41:13.357074+0000","last_peered":"2023-10-06T01:41:13.357074+0000","last_clean":"2023-10-06T01:41:13.357074+0000","last_became_active":"2023-10-06T01:41:12.340598+0000","last_became_peered":"2023-10-06T01:41:12.340598+0000","last_unstale":"2023-10-06T01:41:13.357074+0000","last_undegraded":"2023-10-06T01:41:13.357074+0000","last_fullsized":"2023-10-06T01:41:13.357074+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:06:19.029137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351653+0000","last_change":"2023-10-06T01:41:13.343918+0000","last_active":"2023-10-06T01:41:13.351653+0000","last_peered":"2023-10-06T01:41:13.351653+0000","last_clean":"2023-10-06T01:41:13.351653+0000","last_became_active":"2023-10-06T01:41:13.343621+0000","last_became_peered":"2023-10-06T01:41:13.343621+0000","last_unstale":"2023-10-06T01:41:13.351653+0000","last_undegraded":"2023-10-06T01:41:13.351653+0000","last_fullsized":"2023-10-06T01:41:13.351653+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:42:44.799267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.13","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.350439+0000","last_change":"2023-10-06T01:41:12.343541+0000","last_active":"2023-10-06T01:41:13.350439+0000","last_peered":"2023-10-06T01:41:13.350439+0000","last_clean":"2023-10-06T01:41:13.350439+0000","last_became_active":"2023-10-06T01:41:12.342645+0000","last_became_peered":"2023-10-06T01:41:12.342645+0000","last_unstale":"2023-10-06T01:41:13.350439+0000","last_undegraded":"2023-10-06T01:41:13.350439+0000","last_fullsized":"2023-10-06T01:41:13.350439+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:16:14.826869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.357594+0000","last_change":"2023-10-06T01:41:13.352574+0000","last_active":"2023-10-06T01:41:13.357594+0000","last_peered":"2023-10-06T01:41:13.357594+0000","last_clean":"2023-10-06T01:41:13.357594+0000","last_became_active":"2023-10-06T01:41:13.352300+0000","last_became_peered":"2023-10-06T01:41:13.352300+0000","last_unstale":"2023-10-06T01:41:13.357594+0000","last_undegraded":"2023-10-06T01:41:13.357594+0000","last_fullsized":"2023-10-06T01:41:13.357594+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:27:54.074728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.329224+0000","last_change":"2023-10-06T01:41:12.344679+0000","last_active":"2023-10-06T01:41:13.329224+0000","last_peered":"2023-10-06T01:41:13.329224+0000","last_clean":"2023-10-06T01:41:13.329224+0000","last_became_active":"2023-10-06T01:41:12.344430+0000","last_became_peered":"2023-10-06T01:41:12.344430+0000","last_unstale":"2023-10-06T01:41:13.329224+0000","last_undegraded":"2023-10-06T01:41:13.329224+0000","last_fullsized":"2023-10-06T01:41:13.329224+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:27:01.582307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.360786+0000","last_change":"2023-10-06T01:41:13.357148+0000","last_active":"2023-10-06T01:41:13.360786+0000","last_peered":"2023-10-06T01:41:13.360786+0000","last_clean":"2023-10-06T01:41:13.360786+0000","last_became_active":"2023-10-06T01:41:13.356843+0000","last_became_peered":"2023-10-06T01:41:13.356843+0000","last_unstale":"2023-10-06T01:41:13.360786+0000","last_undegraded":"2023-10-06T01:41:13.360786+0000","last_fullsized":"2023-10-06T01:41:13.360786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T01:51:57.229301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.334167+0000","last_change":"2023-10-06T01:41:12.344068+0000","last_active":"2023-10-06T01:41:13.334167+0000","last_peered":"2023-10-06T01:41:13.334167+0000","last_clean":"2023-10-06T01:41:13.334167+0000","last_became_active":"2023-10-06T01:41:12.342562+0000","last_became_peered":"2023-10-06T01:41:12.342562+0000","last_unstale":"2023-10-06T01:41:13.334167+0000","last_undegraded":"2023-10-06T01:41:13.334167+0000","last_fullsized":"2023-10-06T01:41:13.334167+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:37:13.672716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.346441+0000","last_change":"2023-10-06T01:41:13.343508+0000","last_active":"2023-10-06T01:41:13.346441+0000","last_peered":"2023-10-06T01:41:13.346441+0000","last_clean":"2023-10-06T01:41:13.346441+0000","last_became_active":"2023-10-06T01:41:13.343249+0000","last_became_peered":"2023-10-06T01:41:13.343249+0000","last_unstale":"2023-10-06T01:41:13.346441+0000","last_undegraded":"2023-10-06T01:41:13.346441+0000","last_fullsized":"2023-10-06T01:41:13.346441+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:03:37.385158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"19'8","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.347028+0000","last_change":"2023-10-06T01:41:12.342949+0000","last_active":"2023-10-06T01:41:13.347028+0000","last_peered":"2023-10-06T01:41:13.347028+0000","last_clean":"2023-10-06T01:41:13.347028+0000","last_became_active":"2023-10-06T01:41:12.342687+0000","last_became_peered":"2023-10-06T01:41:12.342687+0000","last_unstale":"2023-10-06T01:41:13.347028+0000","last_undegraded":"2023-10-06T01:41:13.347028+0000","last_fullsized":"2023-10-06T01:41:13.347028+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:37:52.143727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.357895+0000","last_change":"2023-10-06T01:41:13.354861+0000","last_active":"2023-10-06T01:41:13.357895+0000","last_peered":"2023-10-06T01:41:13.357895+0000","last_clean":"2023-10-06T01:41:13.357895+0000","last_became_active":"2023-10-06T01:41:13.354190+0000","last_became_peered":"2023-10-06T01:41:13.354190+0000","last_unstale":"2023-10-06T01:41:13.357895+0000","last_undegraded":"2023-10-06T01:41:13.357895+0000","last_fullsized":"2023-10-06T01:41:13.357895+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:45:07.571069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.f","version":"19'1","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.349986+0000","last_change":"2023-10-06T01:41:12.344048+0000","last_active":"2023-10-06T01:41:13.349986+0000","last_peered":"2023-10-06T01:41:13.349986+0000","last_clean":"2023-10-06T01:41:13.349986+0000","last_became_active":"2023-10-06T01:41:12.343532+0000","last_became_peered":"2023-10-06T01:41:12.343532+0000","last_unstale":"2023-10-06T01:41:13.349986+0000","last_undegraded":"2023-10-06T01:41:13.349986+0000","last_fullsized":"2023-10-06T01:41:13.349986+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:50:44.552768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354457+0000","last_change":"2023-10-06T01:41:13.351276+0000","last_active":"2023-10-06T01:41:13.354457+0000","last_peered":"2023-10-06T01:41:13.354457+0000","last_clean":"2023-10-06T01:41:13.354457+0000","last_became_active":"2023-10-06T01:41:13.351005+0000","last_became_peered":"2023-10-06T01:41:13.351005+0000","last_unstale":"2023-10-06T01:41:13.354457+0000","last_undegraded":"2023-10-06T01:41:13.354457+0000","last_fullsized":"2023-10-06T01:41:13.354457+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:42:51.581939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356755+0000","last_change":"2023-10-06T01:41:12.344159+0000","last_active":"2023-10-06T01:41:13.356755+0000","last_peered":"2023-10-06T01:41:13.356755+0000","last_clean":"2023-10-06T01:41:13.356755+0000","last_became_active":"2023-10-06T01:41:12.343593+0000","last_became_peered":"2023-10-06T01:41:12.343593+0000","last_unstale":"2023-10-06T01:41:13.356755+0000","last_undegraded":"2023-10-06T01:41:13.356755+0000","last_fullsized":"2023-10-06T01:41:13.356755+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:27:01.582307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354635+0000","last_change":"2023-10-06T01:41:13.349120+0000","last_active":"2023-10-06T01:41:13.354635+0000","last_peered":"2023-10-06T01:41:13.354635+0000","last_clean":"2023-10-06T01:41:13.354635+0000","last_became_active":"2023-10-06T01:41:13.348850+0000","last_became_peered":"2023-10-06T01:41:13.348850+0000","last_unstale":"2023-10-06T01:41:13.354635+0000","last_undegraded":"2023-10-06T01:41:13.354635+0000","last_fullsized":"2023-10-06T01:41:13.354635+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:27:32.273170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"19'1","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.358476+0000","last_change":"2023-10-06T01:41:12.344880+0000","last_active":"2023-10-06T01:41:13.358476+0000","last_peered":"2023-10-06T01:41:13.358476+0000","last_clean":"2023-10-06T01:41:13.358476+0000","last_became_active":"2023-10-06T01:41:12.344676+0000","last_became_peered":"2023-10-06T01:41:12.344676+0000","last_unstale":"2023-10-06T01:41:13.358476+0000","last_undegraded":"2023-10-06T01:41:13.358476+0000","last_fullsized":"2023-10-06T01:41:13.358476+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:21:09.833061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.353951+0000","last_change":"2023-10-06T01:41:13.348117+0000","last_active":"2023-10-06T01:41:13.353951+0000","last_peered":"2023-10-06T01:41:13.353951+0000","last_clean":"2023-10-06T01:41:13.353951+0000","last_became_active":"2023-10-06T01:41:13.347872+0000","last_became_peered":"2023-10-06T01:41:13.347872+0000","last_unstale":"2023-10-06T01:41:13.353951+0000","last_undegraded":"2023-10-06T01:41:13.353951+0000","last_fullsized":"2023-10-06T01:41:13.353951+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:21:10.840006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.338677+0000","last_change":"2023-10-06T01:41:12.341362+0000","last_active":"2023-10-06T01:41:13.338677+0000","last_peered":"2023-10-06T01:41:13.338677+0000","last_clean":"2023-10-06T01:41:13.338677+0000","last_became_active":"2023-10-06T01:41:12.340855+0000","last_became_peered":"2023-10-06T01:41:12.340855+0000","last_unstale":"2023-10-06T01:41:13.338677+0000","last_undegraded":"2023-10-06T01:41:13.338677+0000","last_fullsized":"2023-10-06T01:41:13.338677+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:15:40.841990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.349189+0000","last_change":"2023-10-06T01:41:13.344966+0000","last_active":"2023-10-06T01:41:13.349189+0000","last_peered":"2023-10-06T01:41:13.349189+0000","last_clean":"2023-10-06T01:41:13.349189+0000","last_became_active":"2023-10-06T01:41:13.344700+0000","last_became_peered":"2023-10-06T01:41:13.344700+0000","last_unstale":"2023-10-06T01:41:13.349189+0000","last_undegraded":"2023-10-06T01:41:13.349189+0000","last_fullsized":"2023-10-06T01:41:13.349189+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:42:23.199959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.346337+0000","last_change":"2023-10-06T01:41:12.343169+0000","last_active":"2023-10-06T01:41:13.346337+0000","last_peered":"2023-10-06T01:41:13.346337+0000","last_clean":"2023-10-06T01:41:13.346337+0000","last_became_active":"2023-10-06T01:41:12.342474+0000","last_became_peered":"2023-10-06T01:41:12.342474+0000","last_unstale":"2023-10-06T01:41:13.346337+0000","last_undegraded":"2023-10-06T01:41:13.346337+0000","last_fullsized":"2023-10-06T01:41:13.346337+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:27:01.582307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351608+0000","last_change":"2023-10-06T01:41:13.343671+0000","last_active":"2023-10-06T01:41:13.351608+0000","last_peered":"2023-10-06T01:41:13.351608+0000","last_clean":"2023-10-06T01:41:13.351608+0000","last_became_active":"2023-10-06T01:41:13.341868+0000","last_became_peered":"2023-10-06T01:41:13.341868+0000","last_unstale":"2023-10-06T01:41:13.351608+0000","last_undegraded":"2023-10-06T01:41:13.351608+0000","last_fullsized":"2023-10-06T01:41:13.351608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T01:51:57.229301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.329621+0000","last_change":"2023-10-06T01:41:12.339878+0000","last_active":"2023-10-06T01:41:13.329621+0000","last_peered":"2023-10-06T01:41:13.329621+0000","last_clean":"2023-10-06T01:41:13.329621+0000","last_became_active":"2023-10-06T01:41:12.338935+0000","last_became_peered":"2023-10-06T01:41:12.338935+0000","last_unstale":"2023-10-06T01:41:13.329621+0000","last_undegraded":"2023-10-06T01:41:13.329621+0000","last_fullsized":"2023-10-06T01:41:13.329621+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:44:53.014481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351640+0000","last_change":"2023-10-06T01:41:13.344292+0000","last_active":"2023-10-06T01:41:13.351640+0000","last_peered":"2023-10-06T01:41:13.351640+0000","last_clean":"2023-10-06T01:41:13.351640+0000","last_became_active":"2023-10-06T01:41:13.343947+0000","last_became_peered":"2023-10-06T01:41:13.343947+0000","last_unstale":"2023-10-06T01:41:13.351640+0000","last_undegraded":"2023-10-06T01:41:13.351640+0000","last_fullsized":"2023-10-06T01:41:13.351640+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:45:07.571069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.331713+0000","last_change":"2023-10-06T01:41:12.336355+0000","last_active":"2023-10-06T01:41:13.331713+0000","last_peered":"2023-10-06T01:41:13.331713+0000","last_clean":"2023-10-06T01:41:13.331713+0000","last_became_active":"2023-10-06T01:41:12.336064+0000","last_became_peered":"2023-10-06T01:41:12.336064+0000","last_unstale":"2023-10-06T01:41:13.331713+0000","last_undegraded":"2023-10-06T01:41:13.331713+0000","last_fullsized":"2023-10-06T01:41:13.331713+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:16:14.826869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354641+0000","last_change":"2023-10-06T01:41:13.346404+0000","last_active":"2023-10-06T01:41:13.354641+0000","last_peered":"2023-10-06T01:41:13.354641+0000","last_clean":"2023-10-06T01:41:13.354641+0000","last_became_active":"2023-10-06T01:41:13.343805+0000","last_became_peered":"2023-10-06T01:41:13.343805+0000","last_unstale":"2023-10-06T01:41:13.354641+0000","last_undegraded":"2023-10-06T01:41:13.354641+0000","last_fullsized":"2023-10-06T01:41:13.354641+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:11:26.794025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.334388+0000","last_change":"2023-10-06T01:41:12.345661+0000","last_active":"2023-10-06T01:41:13.334388+0000","last_peered":"2023-10-06T01:41:13.334388+0000","last_clean":"2023-10-06T01:41:13.334388+0000","last_became_active":"2023-10-06T01:41:12.345363+0000","last_became_peered":"2023-10-06T01:41:12.345363+0000","last_unstale":"2023-10-06T01:41:13.334388+0000","last_undegraded":"2023-10-06T01:41:13.334388+0000","last_fullsized":"2023-10-06T01:41:13.334388+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:50:44.552768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356650+0000","last_change":"2023-10-06T01:41:13.353633+0000","last_active":"2023-10-06T01:41:13.356650+0000","last_peered":"2023-10-06T01:41:13.356650+0000","last_clean":"2023-10-06T01:41:13.356650+0000","last_became_active":"2023-10-06T01:41:13.353318+0000","last_became_peered":"2023-10-06T01:41:13.353318+0000","last_unstale":"2023-10-06T01:41:13.356650+0000","last_undegraded":"2023-10-06T01:41:13.356650+0000","last_fullsized":"2023-10-06T01:41:13.356650+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:42:23.199959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356615+0000","last_change":"2023-10-06T01:41:12.340463+0000","last_active":"2023-10-06T01:41:13.356615+0000","last_peered":"2023-10-06T01:41:13.356615+0000","last_clean":"2023-10-06T01:41:13.356615+0000","last_became_active":"2023-10-06T01:41:12.340186+0000","last_became_peered":"2023-10-06T01:41:12.340186+0000","last_unstale":"2023-10-06T01:41:13.356615+0000","last_undegraded":"2023-10-06T01:41:13.356615+0000","last_fullsized":"2023-10-06T01:41:13.356615+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:16:14.826869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.330869+0000","last_change":"2023-10-06T01:41:07.454295+0000","last_active":"2023-10-06T01:41:13.330869+0000","last_peered":"2023-10-06T01:41:13.330869+0000","last_clean":"2023-10-06T01:41:13.330869+0000","last_became_active":"2023-10-06T01:41:07.454034+0000","last_became_peered":"2023-10-06T01:41:07.454034+0000","last_unstale":"2023-10-06T01:41:13.330869+0000","last_undegraded":"2023-10-06T01:41:13.330869+0000","last_fullsized":"2023-10-06T01:41:13.330869+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:32:58.914183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354272+0000","last_change":"2023-10-06T01:41:13.351289+0000","last_active":"2023-10-06T01:41:13.354272+0000","last_peered":"2023-10-06T01:41:13.354272+0000","last_clean":"2023-10-06T01:41:13.354272+0000","last_became_active":"2023-10-06T01:41:13.351022+0000","last_became_peered":"2023-10-06T01:41:13.351022+0000","last_unstale":"2023-10-06T01:41:13.354272+0000","last_undegraded":"2023-10-06T01:41:13.354272+0000","last_fullsized":"2023-10-06T01:41:13.354272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T01:51:57.229301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.334655+0000","last_change":"2023-10-06T01:41:07.261955+0000","last_active":"2023-10-06T01:41:13.334655+0000","last_peered":"2023-10-06T01:41:13.334655+0000","last_clean":"2023-10-06T01:41:13.334655+0000","last_became_active":"2023-10-06T01:41:07.261692+0000","last_became_peered":"2023-10-06T01:41:07.261692+0000","last_unstale":"2023-10-06T01:41:13.334655+0000","last_undegraded":"2023-10-06T01:41:13.334655+0000","last_fullsized":"2023-10-06T01:41:13.334655+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"20'1","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-06T01:41:15.784088+0000","last_change":"2023-10-06T01:41:12.341312+0000","last_active":"2023-10-06T01:41:15.784088+0000","last_peered":"2023-10-06T01:41:15.784088+0000","last_clean":"2023-10-06T01:41:15.784088+0000","last_became_active":"2023-10-06T01:41:12.340997+0000","last_became_peered":"2023-10-06T01:41:12.340997+0000","last_unstale":"2023-10-06T01:41:15.784088+0000","last_undegraded":"2023-10-06T01:41:15.784088+0000","last_fullsized":"2023-10-06T01:41:15.784088+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:06:19.029137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351571+0000","last_change":"2023-10-06T01:41:13.344156+0000","last_active":"2023-10-06T01:41:13.351571+0000","last_peered":"2023-10-06T01:41:13.351571+0000","last_clean":"2023-10-06T01:41:13.351571+0000","last_became_active":"2023-10-06T01:41:13.343601+0000","last_became_peered":"2023-10-06T01:41:13.343601+0000","last_unstale":"2023-10-06T01:41:13.351571+0000","last_undegraded":"2023-10-06T01:41:13.351571+0000","last_fullsized":"2023-10-06T01:41:13.351571+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:21:42.851681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.336375+0000","last_change":"2023-10-06T01:41:07.258362+0000","last_active":"2023-10-06T01:41:13.336375+0000","last_peered":"2023-10-06T01:41:13.336375+0000","last_clean":"2023-10-06T01:41:13.336375+0000","last_became_active":"2023-10-06T01:41:07.258055+0000","last_became_peered":"2023-10-06T01:41:07.258055+0000","last_unstale":"2023-10-06T01:41:13.336375+0000","last_undegraded":"2023-10-06T01:41:13.336375+0000","last_fullsized":"2023-10-06T01:41:13.336375+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"16'117","reported_seq":163,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.334859+0000","last_change":"2023-10-06T01:41:07.261164+0000","last_active":"2023-10-06T01:41:13.334859+0000","last_peered":"2023-10-06T01:41:13.334859+0000","last_clean":"2023-10-06T01:41:13.334859+0000","last_became_active":"2023-10-06T01:41:07.260896+0000","last_became_peered":"2023-10-06T01:41:07.260896+0000","last_unstale":"2023-10-06T01:41:13.334859+0000","last_undegraded":"2023-10-06T01:41:13.334859+0000","last_fullsized":"2023-10-06T01:41:13.334859+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:05:22.934401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.328964+0000","last_change":"2023-10-06T01:41:12.343057+0000","last_active":"2023-10-06T01:41:13.328964+0000","last_peered":"2023-10-06T01:41:13.328964+0000","last_clean":"2023-10-06T01:41:13.328964+0000","last_became_active":"2023-10-06T01:41:12.342793+0000","last_became_peered":"2023-10-06T01:41:12.342793+0000","last_unstale":"2023-10-06T01:41:13.328964+0000","last_undegraded":"2023-10-06T01:41:13.328964+0000","last_fullsized":"2023-10-06T01:41:13.328964+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:15:40.841990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351969+0000","last_change":"2023-10-06T01:41:13.346571+0000","last_active":"2023-10-06T01:41:13.351969+0000","last_peered":"2023-10-06T01:41:13.351969+0000","last_clean":"2023-10-06T01:41:13.351969+0000","last_became_active":"2023-10-06T01:41:13.346077+0000","last_became_peered":"2023-10-06T01:41:13.346077+0000","last_unstale":"2023-10-06T01:41:13.351969+0000","last_undegraded":"2023-10-06T01:41:13.351969+0000","last_fullsized":"2023-10-06T01:41:13.351969+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:42:51.581939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.333951+0000","last_change":"2023-10-06T01:41:07.456511+0000","last_active":"2023-10-06T01:41:13.333951+0000","last_peered":"2023-10-06T01:41:13.333951+0000","last_clean":"2023-10-06T01:41:13.333951+0000","last_became_active":"2023-10-06T01:41:07.456024+0000","last_became_peered":"2023-10-06T01:41:07.456024+0000","last_unstale":"2023-10-06T01:41:13.333951+0000","last_undegraded":"2023-10-06T01:41:13.333951+0000","last_fullsized":"2023-10-06T01:41:13.333951+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:51:02.949231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.336635+0000","last_change":"2023-10-06T01:41:12.343382+0000","last_active":"2023-10-06T01:41:13.336635+0000","last_peered":"2023-10-06T01:41:13.336635+0000","last_clean":"2023-10-06T01:41:13.336635+0000","last_became_active":"2023-10-06T01:41:12.343088+0000","last_became_peered":"2023-10-06T01:41:12.343088+0000","last_unstale":"2023-10-06T01:41:13.336635+0000","last_undegraded":"2023-10-06T01:41:13.336635+0000","last_fullsized":"2023-10-06T01:41:13.336635+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:44:53.014481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.350157+0000","last_change":"2023-10-06T01:41:13.346852+0000","last_active":"2023-10-06T01:41:13.350157+0000","last_peered":"2023-10-06T01:41:13.350157+0000","last_clean":"2023-10-06T01:41:13.350157+0000","last_became_active":"2023-10-06T01:41:13.346571+0000","last_became_peered":"2023-10-06T01:41:13.346571+0000","last_unstale":"2023-10-06T01:41:13.350157+0000","last_undegraded":"2023-10-06T01:41:13.350157+0000","last_fullsized":"2023-10-06T01:41:13.350157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:25:04.563235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.334453+0000","last_change":"2023-10-06T01:41:07.456273+0000","last_active":"2023-10-06T01:41:13.334453+0000","last_peered":"2023-10-06T01:41:13.334453+0000","last_clean":"2023-10-06T01:41:13.334453+0000","last_became_active":"2023-10-06T01:41:07.456000+0000","last_became_peered":"2023-10-06T01:41:07.456000+0000","last_unstale":"2023-10-06T01:41:13.334453+0000","last_undegraded":"2023-10-06T01:41:13.334453+0000","last_fullsized":"2023-10-06T01:41:13.334453+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:01:42.633388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.334637+0000","last_change":"2023-10-06T01:41:12.342215+0000","last_active":"2023-10-06T01:41:13.334637+0000","last_peered":"2023-10-06T01:41:13.334637+0000","last_clean":"2023-10-06T01:41:13.334637+0000","last_became_active":"2023-10-06T01:41:12.341493+0000","last_became_peered":"2023-10-06T01:41:12.341493+0000","last_unstale":"2023-10-06T01:41:13.334637+0000","last_undegraded":"2023-10-06T01:41:13.334637+0000","last_fullsized":"2023-10-06T01:41:13.334637+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:49:54.658088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352134+0000","last_change":"2023-10-06T01:41:13.346680+0000","last_active":"2023-10-06T01:41:13.352134+0000","last_peered":"2023-10-06T01:41:13.352134+0000","last_clean":"2023-10-06T01:41:13.352134+0000","last_became_active":"2023-10-06T01:41:13.346326+0000","last_became_peered":"2023-10-06T01:41:13.346326+0000","last_unstale":"2023-10-06T01:41:13.352134+0000","last_undegraded":"2023-10-06T01:41:13.352134+0000","last_fullsized":"2023-10-06T01:41:13.352134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:12:52.741339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.338955+0000","last_change":"2023-10-06T01:41:07.456688+0000","last_active":"2023-10-06T01:41:13.338955+0000","last_peered":"2023-10-06T01:41:13.338955+0000","last_clean":"2023-10-06T01:41:13.338955+0000","last_became_active":"2023-10-06T01:41:07.456304+0000","last_became_peered":"2023-10-06T01:41:07.456304+0000","last_unstale":"2023-10-06T01:41:13.338955+0000","last_undegraded":"2023-10-06T01:41:13.338955+0000","last_fullsized":"2023-10-06T01:41:13.338955+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.332203+0000","last_change":"2023-10-06T01:41:12.342354+0000","last_active":"2023-10-06T01:41:13.332203+0000","last_peered":"2023-10-06T01:41:13.332203+0000","last_clean":"2023-10-06T01:41:13.332203+0000","last_became_active":"2023-10-06T01:41:12.342066+0000","last_became_peered":"2023-10-06T01:41:12.342066+0000","last_unstale":"2023-10-06T01:41:13.332203+0000","last_undegraded":"2023-10-06T01:41:13.332203+0000","last_fullsized":"2023-10-06T01:41:13.332203+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:15:40.841990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352905+0000","last_change":"2023-10-06T01:41:13.348241+0000","last_active":"2023-10-06T01:41:13.352905+0000","last_peered":"2023-10-06T01:41:13.352905+0000","last_clean":"2023-10-06T01:41:13.352905+0000","last_became_active":"2023-10-06T01:41:13.347979+0000","last_became_peered":"2023-10-06T01:41:13.347979+0000","last_unstale":"2023-10-06T01:41:13.352905+0000","last_undegraded":"2023-10-06T01:41:13.352905+0000","last_fullsized":"2023-10-06T01:41:13.352905+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:41:59.186469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.329416+0000","last_change":"2023-10-06T01:41:07.456388+0000","last_active":"2023-10-06T01:41:13.329416+0000","last_peered":"2023-10-06T01:41:13.329416+0000","last_clean":"2023-10-06T01:41:13.329416+0000","last_became_active":"2023-10-06T01:41:07.456045+0000","last_became_peered":"2023-10-06T01:41:07.456045+0000","last_unstale":"2023-10-06T01:41:13.329416+0000","last_undegraded":"2023-10-06T01:41:13.329416+0000","last_fullsized":"2023-10-06T01:41:13.329416+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.338387+0000","last_change":"2023-10-06T01:41:12.344268+0000","last_active":"2023-10-06T01:41:13.338387+0000","last_peered":"2023-10-06T01:41:13.338387+0000","last_clean":"2023-10-06T01:41:13.338387+0000","last_became_active":"2023-10-06T01:41:12.343996+0000","last_became_peered":"2023-10-06T01:41:12.343996+0000","last_unstale":"2023-10-06T01:41:13.338387+0000","last_undegraded":"2023-10-06T01:41:13.338387+0000","last_fullsized":"2023-10-06T01:41:13.338387+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:37:52.143727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351761+0000","last_change":"2023-10-06T01:41:13.346184+0000","last_active":"2023-10-06T01:41:13.351761+0000","last_peered":"2023-10-06T01:41:13.351761+0000","last_clean":"2023-10-06T01:41:13.351761+0000","last_became_active":"2023-10-06T01:41:13.345570+0000","last_became_peered":"2023-10-06T01:41:13.345570+0000","last_unstale":"2023-10-06T01:41:13.351761+0000","last_undegraded":"2023-10-06T01:41:13.351761+0000","last_fullsized":"2023-10-06T01:41:13.351761+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:42:46.214805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.330470+0000","last_change":"2023-10-06T01:41:07.258162+0000","last_active":"2023-10-06T01:41:13.330470+0000","last_peered":"2023-10-06T01:41:13.330470+0000","last_clean":"2023-10-06T01:41:13.330470+0000","last_became_active":"2023-10-06T01:41:07.257795+0000","last_became_peered":"2023-10-06T01:41:07.257795+0000","last_unstale":"2023-10-06T01:41:13.330470+0000","last_undegraded":"2023-10-06T01:41:13.330470+0000","last_fullsized":"2023-10-06T01:41:13.330470+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:05:22.934401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":10,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.337002+0000","last_change":"2023-10-06T01:41:12.341990+0000","last_active":"2023-10-06T01:41:13.337002+0000","last_peered":"2023-10-06T01:41:13.337002+0000","last_clean":"2023-10-06T01:41:13.337002+0000","last_became_active":"2023-10-06T01:41:12.341680+0000","last_became_peered":"2023-10-06T01:41:12.341680+0000","last_unstale":"2023-10-06T01:41:13.337002+0000","last_undegraded":"2023-10-06T01:41:13.337002+0000","last_fullsized":"2023-10-06T01:41:13.337002+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:50:44.552768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.349056+0000","last_change":"2023-10-06T01:41:13.344589+0000","last_active":"2023-10-06T01:41:13.349056+0000","last_peered":"2023-10-06T01:41:13.349056+0000","last_clean":"2023-10-06T01:41:13.349056+0000","last_became_active":"2023-10-06T01:41:13.344281+0000","last_became_peered":"2023-10-06T01:41:13.344281+0000","last_unstale":"2023-10-06T01:41:13.349056+0000","last_undegraded":"2023-10-06T01:41:13.349056+0000","last_fullsized":"2023-10-06T01:41:13.349056+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:32:17.938309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"19'1","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.345810+0000","last_change":"2023-10-06T01:41:12.347655+0000","last_active":"2023-10-06T01:41:13.345810+0000","last_peered":"2023-10-06T01:41:13.345810+0000","last_clean":"2023-10-06T01:41:13.345810+0000","last_became_active":"2023-10-06T01:41:12.347353+0000","last_became_peered":"2023-10-06T01:41:12.347353+0000","last_unstale":"2023-10-06T01:41:13.345810+0000","last_undegraded":"2023-10-06T01:41:13.345810+0000","last_fullsized":"2023-10-06T01:41:13.345810+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:49:54.658088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352446+0000","last_change":"2023-10-06T01:41:13.349874+0000","last_active":"2023-10-06T01:41:13.352446+0000","last_peered":"2023-10-06T01:41:13.352446+0000","last_clean":"2023-10-06T01:41:13.352446+0000","last_became_active":"2023-10-06T01:41:13.349607+0000","last_became_peered":"2023-10-06T01:41:13.349607+0000","last_unstale":"2023-10-06T01:41:13.352446+0000","last_undegraded":"2023-10-06T01:41:13.352446+0000","last_fullsized":"2023-10-06T01:41:13.352446+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:06:20.036081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1c","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.352575+0000","last_change":"2023-10-06T01:41:12.341511+0000","last_active":"2023-10-06T01:41:13.352575+0000","last_peered":"2023-10-06T01:41:13.352575+0000","last_clean":"2023-10-06T01:41:13.352575+0000","last_became_active":"2023-10-06T01:41:12.341216+0000","last_became_peered":"2023-10-06T01:41:12.341216+0000","last_unstale":"2023-10-06T01:41:13.352575+0000","last_undegraded":"2023-10-06T01:41:13.352575+0000","last_fullsized":"2023-10-06T01:41:13.352575+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:27:01.582307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351635+0000","last_change":"2023-10-06T01:41:13.343780+0000","last_active":"2023-10-06T01:41:13.351635+0000","last_peered":"2023-10-06T01:41:13.351635+0000","last_clean":"2023-10-06T01:41:13.351635+0000","last_became_active":"2023-10-06T01:41:13.343515+0000","last_became_peered":"2023-10-06T01:41:13.343515+0000","last_unstale":"2023-10-06T01:41:13.351635+0000","last_undegraded":"2023-10-06T01:41:13.351635+0000","last_fullsized":"2023-10-06T01:41:13.351635+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:38:58.407819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"19'2","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.353702+0000","last_change":"2023-10-06T01:41:12.341780+0000","last_active":"2023-10-06T01:41:13.353702+0000","last_peered":"2023-10-06T01:41:13.353702+0000","last_clean":"2023-10-06T01:41:13.353702+0000","last_became_active":"2023-10-06T01:41:12.340585+0000","last_became_peered":"2023-10-06T01:41:12.340585+0000","last_unstale":"2023-10-06T01:41:13.353702+0000","last_undegraded":"2023-10-06T01:41:13.353702+0000","last_fullsized":"2023-10-06T01:41:13.353702+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:49:54.658088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354558+0000","last_change":"2023-10-06T01:41:13.345543+0000","last_active":"2023-10-06T01:41:13.354558+0000","last_peered":"2023-10-06T01:41:13.354558+0000","last_clean":"2023-10-06T01:41:13.354558+0000","last_became_active":"2023-10-06T01:41:13.345286+0000","last_became_peered":"2023-10-06T01:41:13.345286+0000","last_unstale":"2023-10-06T01:41:13.354558+0000","last_undegraded":"2023-10-06T01:41:13.354558+0000","last_fullsized":"2023-10-06T01:41:13.354558+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:52:48.286318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"19'1","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351829+0000","last_change":"2023-10-06T01:41:12.340776+0000","last_active":"2023-10-06T01:41:13.351829+0000","last_peered":"2023-10-06T01:41:13.351829+0000","last_clean":"2023-10-06T01:41:13.351829+0000","last_became_active":"2023-10-06T01:41:12.340294+0000","last_became_peered":"2023-10-06T01:41:12.340294+0000","last_unstale":"2023-10-06T01:41:13.351829+0000","last_undegraded":"2023-10-06T01:41:13.351829+0000","last_fullsized":"2023-10-06T01:41:13.351829+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:37:13.672716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.357969+0000","last_change":"2023-10-06T01:41:13.355273+0000","last_active":"2023-10-06T01:41:13.357969+0000","last_peered":"2023-10-06T01:41:13.357969+0000","last_clean":"2023-10-06T01:41:13.357969+0000","last_became_active":"2023-10-06T01:41:13.354977+0000","last_became_peered":"2023-10-06T01:41:13.354977+0000","last_unstale":"2023-10-06T01:41:13.357969+0000","last_undegraded":"2023-10-06T01:41:13.357969+0000","last_fullsized":"2023-10-06T01:41:13.357969+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:21:42.851681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"19'4","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.345580+0000","last_change":"2023-10-06T01:41:12.338431+0000","last_active":"2023-10-06T01:41:13.345580+0000","last_peered":"2023-10-06T01:41:13.345580+0000","last_clean":"2023-10-06T01:41:13.345580+0000","last_became_active":"2023-10-06T01:41:12.338081+0000","last_became_peered":"2023-10-06T01:41:12.338081+0000","last_unstale":"2023-10-06T01:41:13.345580+0000","last_undegraded":"2023-10-06T01:41:13.345580+0000","last_fullsized":"2023-10-06T01:41:13.345580+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:11.306723+0000","last_clean_scrub_stamp":"2023-10-06T01:41:11.306723+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:16:14.826869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351157+0000","last_change":"2023-10-06T01:41:13.342212+0000","last_active":"2023-10-06T01:41:13.351157+0000","last_peered":"2023-10-06T01:41:13.351157+0000","last_clean":"2023-10-06T01:41:13.351157+0000","last_became_active":"2023-10-06T01:41:13.341948+0000","last_became_peered":"2023-10-06T01:41:13.341948+0000","last_unstale":"2023-10-06T01:41:13.351157+0000","last_undegraded":"2023-10-06T01:41:13.351157+0000","last_fullsized":"2023-10-06T01:41:13.351157+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:18:23.688664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.20","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.354188+0000","last_change":"2023-10-06T01:41:13.348482+0000","last_active":"2023-10-06T01:41:13.354188+0000","last_peered":"2023-10-06T01:41:13.354188+0000","last_clean":"2023-10-06T01:41:13.354188+0000","last_became_active":"2023-10-06T01:41:13.348075+0000","last_became_peered":"2023-10-06T01:41:13.348075+0000","last_unstale":"2023-10-06T01:41:13.354188+0000","last_undegraded":"2023-10-06T01:41:13.354188+0000","last_fullsized":"2023-10-06T01:41:13.354188+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:25:04.563235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.21","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.357958+0000","last_change":"2023-10-06T01:41:13.355216+0000","last_active":"2023-10-06T01:41:13.357958+0000","last_peered":"2023-10-06T01:41:13.357958+0000","last_clean":"2023-10-06T01:41:13.357958+0000","last_became_active":"2023-10-06T01:41:13.354853+0000","last_became_peered":"2023-10-06T01:41:13.354853+0000","last_unstale":"2023-10-06T01:41:13.357958+0000","last_undegraded":"2023-10-06T01:41:13.357958+0000","last_fullsized":"2023-10-06T01:41:13.357958+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:42:44.799267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.22","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.351163+0000","last_change":"2023-10-06T01:41:13.342888+0000","last_active":"2023-10-06T01:41:13.351163+0000","last_peered":"2023-10-06T01:41:13.351163+0000","last_clean":"2023-10-06T01:41:13.351163+0000","last_became_active":"2023-10-06T01:41:13.342626+0000","last_became_peered":"2023-10-06T01:41:13.342626+0000","last_unstale":"2023-10-06T01:41:13.351163+0000","last_undegraded":"2023-10-06T01:41:13.351163+0000","last_fullsized":"2023-10-06T01:41:13.351163+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:43:29.491858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.23","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.357705+0000","last_change":"2023-10-06T01:41:13.353822+0000","last_active":"2023-10-06T01:41:13.357705+0000","last_peered":"2023-10-06T01:41:13.357705+0000","last_clean":"2023-10-06T01:41:13.357705+0000","last_became_active":"2023-10-06T01:41:13.353292+0000","last_became_peered":"2023-10-06T01:41:13.353292+0000","last_unstale":"2023-10-06T01:41:13.357705+0000","last_undegraded":"2023-10-06T01:41:13.357705+0000","last_fullsized":"2023-10-06T01:41:13.357705+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:52:48.286318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.24","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-06T01:41:13.356636+0000","last_change":"2023-10-06T01:41:13.353662+0000","last_active":"2023-10-06T01:41:13.356636+0000","last_peered":"2023-10-06T01:41:13.356636+0000","last_clean":"2023-10-06T01:41:13.356636+0000","last_became_active":"2023-10-06T01:41:13.353269+0000","last_became_peered":"2023-10-06T01:41:13.353269+0000","last_unstale":"2023-10-06T01:41:13.356636+0000","last_undegraded":"2023-10-06T01:41:13.356636+0000","last_fullsized":"2023-10-06T01:41:13.356636+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:12.313668+0000","last_clean_scrub_stamp":"2023-10-06T01:41:12.313668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:11:44.715827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574857,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298096,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073744,"statfs":{"total":96636764160,"available":96331513856,"internally_reserved":0,"allocated":827392,"data_stored":666304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70699999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.505}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297484,"kb_used_data":196,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074356,"statfs":{"total":96636764160,"available":96332140544,"internally_reserved":0,"allocated":200704,"data_stored":72831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298068,"kb_used_data":780,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073772,"statfs":{"total":96636764160,"available":96331542528,"internally_reserved":0,"allocated":798720,"data_stored":664552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297488,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074352,"statfs":{"total":96636764160,"available":96332136448,"internally_reserved":0,"allocated":204800,"data_stored":73385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.5}]}]},{"osd":2,"up_from":13,"seq":55834574856,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297488,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074352,"statfs":{"total":96636764160,"available":96332136448,"internally_reserved":0,"allocated":204800,"data_stored":74184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.308}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29699999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45300000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297488,"kb_used_data":200,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074352,"statfs":{"total":96636764160,"available":96332136448,"internally_reserved":0,"allocated":204800,"data_stored":72921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.621}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64000000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297508,"kb_used_data":220,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074332,"statfs":{"total":96636764160,"available":96332115968,"internally_reserved":0,"allocated":225280,"data_stored":75776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84999999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97899999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297492,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074348,"statfs":{"total":96636764160,"available":96332132352,"internally_reserved":0,"allocated":208896,"data_stored":73455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.372}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40000000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-06T01:41:26.580 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-06T01:41:26.580 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-06T01:41:26.580 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-06T01:41:26.581 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-06T01:41:26.732 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.730+0000 7f7c53263700 1 Processor -- start 2023-10-06T01:41:26.732 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.730+0000 7f7c53263700 1 -- start start 2023-10-06T01:41:26.734 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.731+0000 7f7c53263700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7c4c1595d0 0x7f7c4c0b32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:26.734 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.731+0000 7f7c53263700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7c4c068640 0x7f7c4c068aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:26.734 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.731+0000 7f7c53263700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c4c0b54b0 con 0x7f7c4c0b1230 2023-10-06T01:41:26.734 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.731+0000 7f7c53263700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c4c0b1c10 con 0x7f7c4c1595d0 2023-10-06T01:41:26.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.731+0000 7f7c53263700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4c0b16a0 con 0x7f7c4c068640 2023-10-06T01:41:26.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.731+0000 7f7c50fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7c4c1595d0 0x7f7c4c0b32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:26.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.731+0000 7f7c4bfff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7c4c0b1230 0x7f7c4c0b5940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57032/0 (socket says 172.21.15.27:57032) 2023-10-06T01:41:26.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.731+0000 7f7c50fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7c4c1595d0 0x7f7c4c0b32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52286/0 (socket says 172.21.15.27:52286) 2023-10-06T01:41:26.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.731+0000 7f7c4bfff700 1 -- 172.21.15.27:0/2888570321 learned_addr learned my addr 172.21.15.27:0/2888570321 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:26.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.732+0000 7f7c51800700 1 --2- 172.21.15.27:0/2888570321 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7c4c068640 0x7f7c4c068aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:26.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.732+0000 7f7c50fff700 1 -- 172.21.15.27:0/2888570321 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7c4c068640 msgr2=0x7f7c4c068aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:26.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.732+0000 7f7c50fff700 1 --2- 172.21.15.27:0/2888570321 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7c4c068640 0x7f7c4c068aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:26.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.732+0000 7f7c50fff700 1 -- 172.21.15.27:0/2888570321 >> v1:172.21.15.27:6789/0 conn(0x7f7c4c0b1230 legacy=0x7f7c4c0b5940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:26.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.732+0000 7f7c50fff700 1 -- 172.21.15.27:0/2888570321 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c4c0b4c00 con 0x7f7c4c1595d0 2023-10-06T01:41:26.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.732+0000 7f7c51800700 1 --2- 172.21.15.27:0/2888570321 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7c4c068640 0x7f7c4c068aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:41:26.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.732+0000 7f7c4b7fe700 1 -- 172.21.15.27:0/2888570321 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3785325498 0 0) 0x7f7c4c0b54b0 con 0x7f7c4c0b1230 2023-10-06T01:41:26.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.732+0000 7f7c50fff700 1 --2- 172.21.15.27:0/2888570321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c4c1595d0 0x7f7c4c0b32f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7c400098e0 tx=0x7f7c400091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cff7bc67e31b075 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:26.736 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.732+0000 7f7c4b7fe700 1 -- 172.21.15.27:0/2888570321 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c40009e80 con 0x7f7c4c1595d0 2023-10-06T01:41:26.736 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.732+0000 7f7c4b7fe700 1 -- 172.21.15.27:0/2888570321 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c4000b860 con 0x7f7c4c1595d0 2023-10-06T01:41:26.736 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.732+0000 7f7c4b7fe700 1 -- 172.21.15.27:0/2888570321 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4000bbe0 con 0x7f7c4c1595d0 2023-10-06T01:41:26.736 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.733+0000 7f7c53263700 1 -- 172.21.15.27:0/2888570321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c4c1595d0 msgr2=0x7f7c4c0b32f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:26.736 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.733+0000 7f7c53263700 1 --2- 172.21.15.27:0/2888570321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c4c1595d0 0x7f7c4c0b32f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7c400098e0 tx=0x7f7c400091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:26.736 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.733+0000 7f7c53263700 1 -- 172.21.15.27:0/2888570321 shutdown_connections 2023-10-06T01:41:26.736 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.733+0000 7f7c53263700 1 --2- 172.21.15.27:0/2888570321 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7c4c068640 0x7f7c4c068aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:26.736 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.733+0000 7f7c53263700 1 --2- 172.21.15.27:0/2888570321 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c4c1595d0 0x7f7c4c0b32f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:26.736 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.733+0000 7f7c53263700 1 -- 172.21.15.27:0/2888570321 >> 172.21.15.27:0/2888570321 conn(0x7f7c4c0ccfa0 msgr2=0x7f7c4c0cd370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:26.736 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.733+0000 7f7c53263700 1 -- 172.21.15.27:0/2888570321 shutdown_connections 2023-10-06T01:41:26.736 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.733+0000 7f7c53263700 1 -- 172.21.15.27:0/2888570321 wait complete. 2023-10-06T01:41:26.738 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.734+0000 7f7c53263700 1 Processor -- start 2023-10-06T01:41:26.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.734+0000 7f7c53263700 1 -- start start 2023-10-06T01:41:26.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.735+0000 7f7c53263700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7c4c068640 0x7f7c4c14dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:26.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.735+0000 7f7c53263700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c4c0b1230 0x7f7c4c16a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:26.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.735+0000 7f7c53263700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7c4c1595d0 0x7f7c4c151ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:26.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.735+0000 7f7c53263700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4c15aa30 con 0x7f7c4c0b1230 2023-10-06T01:41:26.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.735+0000 7f7c53263700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4c159b40 con 0x7f7c4c068640 2023-10-06T01:41:26.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.735+0000 7f7c53263700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4c159e20 con 0x7f7c4c1595d0 2023-10-06T01:41:26.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.735+0000 7f7c4bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c4c0b1230 0x7f7c4c16a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:26.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.735+0000 7f7c4bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c4c0b1230 0x7f7c4c16a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52288/0 (socket says 172.21.15.27:52288) 2023-10-06T01:41:26.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.735+0000 7f7c4bfff700 1 -- 172.21.15.27:0/3229259314 learned_addr learned my addr 172.21.15.27:0/3229259314 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:26.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.735+0000 7f7c51800700 1 --2- 172.21.15.27:0/3229259314 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7c4c1595d0 0x7f7c4c151ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:26.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.735+0000 7f7c50fff700 1 --2- 172.21.15.27:0/3229259314 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7c4c068640 0x7f7c4c14dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:26.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.735+0000 7f7c4bfff700 1 -- 172.21.15.27:0/3229259314 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7c4c1595d0 msgr2=0x7f7c4c151ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:26.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.735+0000 7f7c4bfff700 1 --2- 172.21.15.27:0/3229259314 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7c4c1595d0 0x7f7c4c151ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:26.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.735+0000 7f7c4bfff700 1 -- 172.21.15.27:0/3229259314 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7c4c068640 msgr2=0x7f7c4c14dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:26.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.735+0000 7f7c4bfff700 1 --2- 172.21.15.27:0/3229259314 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7c4c068640 0x7f7c4c14dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:26.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.735+0000 7f7c4bfff700 1 -- 172.21.15.27:0/3229259314 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c40009420 con 0x7f7c4c0b1230 2023-10-06T01:41:26.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.735+0000 7f7c4bfff700 1 --2- 172.21.15.27:0/3229259314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c4c0b1230 0x7f7c4c16a030 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7c3800c7e0 tx=0x7f7c38007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:26.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.736+0000 7f7c48ff9700 1 -- 172.21.15.27:0/3229259314 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c38005b30 con 0x7f7c4c0b1230 2023-10-06T01:41:26.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.736+0000 7f7c48ff9700 1 -- 172.21.15.27:0/3229259314 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c38005c90 con 0x7f7c4c0b1230 2023-10-06T01:41:26.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.736+0000 7f7c53263700 1 -- 172.21.15.27:0/3229259314 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c4c15a100 con 0x7f7c4c0b1230 2023-10-06T01:41:26.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.736+0000 7f7c48ff9700 1 -- 172.21.15.27:0/3229259314 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c38007300 con 0x7f7c4c0b1230 2023-10-06T01:41:26.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.736+0000 7f7c53263700 1 -- 172.21.15.27:0/3229259314 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c4c15a6c0 con 0x7f7c4c0b1230 2023-10-06T01:41:26.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.737+0000 7f7c53263700 1 -- 172.21.15.27:0/3229259314 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c4c15aba0 con 0x7f7c4c0b1230 2023-10-06T01:41:26.741 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.737+0000 7f7c48ff9700 1 -- 172.21.15.27:0/3229259314 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7c3801b3b0 con 0x7f7c4c0b1230 2023-10-06T01:41:26.741 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.737+0000 7f7c48ff9700 1 --2- 172.21.15.27:0/3229259314 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7c30066f30 0x7f7c300693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:26.741 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.738+0000 7f7c48ff9700 1 -- 172.21.15.27:0/3229259314 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f7c38083bb0 con 0x7f7c4c0b1230 2023-10-06T01:41:26.741 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.738+0000 7f7c50fff700 1 --2- 172.21.15.27:0/3229259314 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7c30066f30 0x7f7c300693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:26.741 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.738+0000 7f7c50fff700 1 --2- 172.21.15.27:0/3229259314 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7c30066f30 0x7f7c300693e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7c400091e0 tx=0x7f7c40017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:26.743 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.742+0000 7f7c48ff9700 1 -- 172.21.15.27:0/3229259314 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7c38050cc0 con 0x7f7c4c0b1230 2023-10-06T01:41:26.980 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.978+0000 7f7c53263700 1 -- 172.21.15.27:0/3229259314 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7c4c04c3c0 con 0x7f7c4c0b1230 2023-10-06T01:41:26.980 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.979+0000 7f7c48ff9700 1 -- 172.21.15.27:0/3229259314 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f7c38017080 con 0x7f7c4c0b1230 2023-10-06T01:41:26.980 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:26.980 INFO:teuthology.orchestra.run.smithi027.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-06T01:41:26.982 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.980+0000 7f7c53263700 1 -- 172.21.15.27:0/3229259314 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7c30066f30 msgr2=0x7f7c300693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:26.982 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.980+0000 7f7c53263700 1 --2- 172.21.15.27:0/3229259314 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7c30066f30 0x7f7c300693e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7c400091e0 tx=0x7f7c40017000 comp rx=0 tx=0).stop 2023-10-06T01:41:26.982 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.980+0000 7f7c53263700 1 -- 172.21.15.27:0/3229259314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c4c0b1230 msgr2=0x7f7c4c16a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:26.982 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.980+0000 7f7c53263700 1 --2- 172.21.15.27:0/3229259314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c4c0b1230 0x7f7c4c16a030 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7c3800c7e0 tx=0x7f7c38007df0 comp rx=0 tx=0).stop 2023-10-06T01:41:26.982 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.981+0000 7f7c53263700 1 -- 172.21.15.27:0/3229259314 shutdown_connections 2023-10-06T01:41:26.982 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.981+0000 7f7c53263700 1 --2- 172.21.15.27:0/3229259314 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7c30066f30 0x7f7c300693e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:26.982 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.981+0000 7f7c53263700 1 --2- 172.21.15.27:0/3229259314 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7c4c1595d0 0x7f7c4c151ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:26.983 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.981+0000 7f7c53263700 1 --2- 172.21.15.27:0/3229259314 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7c4c068640 0x7f7c4c14dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:26.983 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.981+0000 7f7c53263700 1 --2- 172.21.15.27:0/3229259314 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7c4c0b1230 0x7f7c4c16a030 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:26.983 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.981+0000 7f7c53263700 1 -- 172.21.15.27:0/3229259314 >> 172.21.15.27:0/3229259314 conn(0x7f7c4c0ccfa0 msgr2=0x7f7c4c156410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:26.983 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.981+0000 7f7c53263700 1 -- 172.21.15.27:0/3229259314 shutdown_connections 2023-10-06T01:41:26.983 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:26.981+0000 7f7c53263700 1 -- 172.21.15.27:0/3229259314 wait complete. 2023-10-06T01:41:26.994 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-06T01:41:26.994 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T01:41:27.112 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.111+0000 7f1f772ca700 1 Processor -- start 2023-10-06T01:41:27.113 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.111+0000 7f1f772ca700 1 -- start start 2023-10-06T01:41:27.115 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.112+0000 7f1f772ca700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1f700cdc00 0x7f1f700cbe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:27.115 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.112+0000 7f1f772ca700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1f700d1030 0x7f1f700c66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:27.115 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.112+0000 7f1f772ca700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f1f700c6bf0 0x7f1f700ae5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:27.115 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.112+0000 7f1f772ca700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f700cc390 con 0x7f1f700cdc00 2023-10-06T01:41:27.116 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.112+0000 7f1f772ca700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f700c7070 con 0x7f1f700c6bf0 2023-10-06T01:41:27.116 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.112+0000 7f1f772ca700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f700ae140 con 0x7f1f700d1030 2023-10-06T01:41:27.116 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.112+0000 7f1f75066700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1f700cdc00 0x7f1f700cbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:27.116 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.112+0000 7f1f75066700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1f700cdc00 0x7f1f700cbe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52304/0 (socket says 172.21.15.27:52304) 2023-10-06T01:41:27.116 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.112+0000 7f1f75066700 1 -- 172.21.15.27:0/822397100 learned_addr learned my addr 172.21.15.27:0/822397100 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:27.116 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.113+0000 7f1f75867700 1 --2- 172.21.15.27:0/822397100 >> v2:172.21.15.124:3300/0 conn(0x7f1f700c6bf0 0x7f1f700ae5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:27.116 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.113+0000 7f1f74865700 1 --2- 172.21.15.27:0/822397100 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1f700d1030 0x7f1f700c66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:27.116 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.113+0000 7f1f75066700 1 -- 172.21.15.27:0/822397100 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1f700d1030 msgr2=0x7f1f700c66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:27.116 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.113+0000 7f1f75066700 1 --2- 172.21.15.27:0/822397100 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1f700d1030 0x7f1f700c66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:27.116 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.113+0000 7f1f75066700 1 -- 172.21.15.27:0/822397100 >> v2:172.21.15.124:3300/0 conn(0x7f1f700c6bf0 msgr2=0x7f1f700ae5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:27.116 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.113+0000 7f1f75066700 1 --2- 172.21.15.27:0/822397100 >> v2:172.21.15.124:3300/0 conn(0x7f1f700c6bf0 0x7f1f700ae5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:27.117 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.113+0000 7f1f75066700 1 -- 172.21.15.27:0/822397100 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f700ad530 con 0x7f1f700cdc00 2023-10-06T01:41:27.117 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.113+0000 7f1f74865700 1 --2- 172.21.15.27:0/822397100 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1f700d1030 0x7f1f700c66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:41:27.117 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.113+0000 7f1f75066700 1 --2- 172.21.15.27:0/822397100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f700cdc00 0x7f1f700cbe50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1f700ae0b0 tx=0x7f1f68009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abadc6e50d056a7c server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:27.117 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.113+0000 7f1f6ffff700 1 -- 172.21.15.27:0/822397100 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f68005c10 con 0x7f1f700cdc00 2023-10-06T01:41:27.117 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.113+0000 7f1f6ffff700 1 -- 172.21.15.27:0/822397100 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f68005d70 con 0x7f1f700cdc00 2023-10-06T01:41:27.117 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.113+0000 7f1f6ffff700 1 -- 172.21.15.27:0/822397100 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f6800b840 con 0x7f1f700cdc00 2023-10-06T01:41:27.117 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.113+0000 7f1f772ca700 1 -- 172.21.15.27:0/822397100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f700cdc00 msgr2=0x7f1f700cbe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:27.117 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.113+0000 7f1f772ca700 1 --2- 172.21.15.27:0/822397100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f700cdc00 0x7f1f700cbe50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1f700ae0b0 tx=0x7f1f68009220 comp rx=0 tx=0).stop 2023-10-06T01:41:27.117 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.113+0000 7f1f772ca700 1 -- 172.21.15.27:0/822397100 shutdown_connections 2023-10-06T01:41:27.117 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.113+0000 7f1f772ca700 1 --2- 172.21.15.27:0/822397100 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f700cdc00 0x7f1f700cbe50 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:27.117 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.113+0000 7f1f772ca700 1 --2- 172.21.15.27:0/822397100 >> v2:172.21.15.124:3300/0 conn(0x7f1f700c6bf0 0x7f1f700ae5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:27.117 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.113+0000 7f1f772ca700 1 --2- 172.21.15.27:0/822397100 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1f700d1030 0x7f1f700c66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:27.118 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.113+0000 7f1f772ca700 1 -- 172.21.15.27:0/822397100 >> 172.21.15.27:0/822397100 conn(0x7f1f70099990 msgr2=0x7f1f700b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:27.118 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.114+0000 7f1f772ca700 1 -- 172.21.15.27:0/822397100 shutdown_connections 2023-10-06T01:41:27.118 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.114+0000 7f1f772ca700 1 -- 172.21.15.27:0/822397100 wait complete. 2023-10-06T01:41:27.120 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.115+0000 7f1f772ca700 1 Processor -- start 2023-10-06T01:41:27.121 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.115+0000 7f1f772ca700 1 -- start start 2023-10-06T01:41:27.121 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.115+0000 7f1f772ca700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f700c6bf0 0x7f1f701582f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:27.121 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.116+0000 7f1f772ca700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1f700cdc00 0x7f1f700ce5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:27.121 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.116+0000 7f1f772ca700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1f700d1030 0x7f1f700ceb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:27.121 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.116+0000 7f1f772ca700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f70169910 con 0x7f1f700c6bf0 2023-10-06T01:41:27.121 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.116+0000 7f1f772ca700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f70158830 con 0x7f1f700d1030 2023-10-06T01:41:27.121 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.116+0000 7f1f772ca700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7009bcf0 con 0x7f1f700cdc00 2023-10-06T01:41:27.121 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.116+0000 7f1f75066700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f700c6bf0 0x7f1f701582f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:27.121 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.116+0000 7f1f75066700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f700c6bf0 0x7f1f701582f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52308/0 (socket says 172.21.15.27:52308) 2023-10-06T01:41:27.121 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.116+0000 7f1f75066700 1 -- 172.21.15.27:0/4213697154 learned_addr learned my addr 172.21.15.27:0/4213697154 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:27.121 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.116+0000 7f1f75867700 1 --2- 172.21.15.27:0/4213697154 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1f700d1030 0x7f1f700ceb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:27.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.116+0000 7f1f74865700 1 --2- 172.21.15.27:0/4213697154 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1f700cdc00 0x7f1f700ce5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:27.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.116+0000 7f1f75066700 1 -- 172.21.15.27:0/4213697154 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1f700cdc00 msgr2=0x7f1f700ce5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:27.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.116+0000 7f1f75066700 1 --2- 172.21.15.27:0/4213697154 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1f700cdc00 0x7f1f700ce5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:27.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.116+0000 7f1f75066700 1 -- 172.21.15.27:0/4213697154 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1f700d1030 msgr2=0x7f1f700ceb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:27.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.116+0000 7f1f75066700 1 --2- 172.21.15.27:0/4213697154 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1f700d1030 0x7f1f700ceb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:27.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.116+0000 7f1f75066700 1 -- 172.21.15.27:0/4213697154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f68009480 con 0x7f1f700c6bf0 2023-10-06T01:41:27.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.116+0000 7f1f75066700 1 --2- 172.21.15.27:0/4213697154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f700c6bf0 0x7f1f701582f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1f68009260 tx=0x7f1f68016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:27.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.116+0000 7f1f6d7fa700 1 -- 172.21.15.27:0/4213697154 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f68006b60 con 0x7f1f700c6bf0 2023-10-06T01:41:27.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.116+0000 7f1f6d7fa700 1 -- 172.21.15.27:0/4213697154 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f68011880 con 0x7f1f700c6bf0 2023-10-06T01:41:27.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.116+0000 7f1f772ca700 1 -- 172.21.15.27:0/4213697154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f7009bfd0 con 0x7f1f700c6bf0 2023-10-06T01:41:27.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.116+0000 7f1f6d7fa700 1 -- 172.21.15.27:0/4213697154 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f68011c20 con 0x7f1f700c6bf0 2023-10-06T01:41:27.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.116+0000 7f1f772ca700 1 -- 172.21.15.27:0/4213697154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f7009c590 con 0x7f1f700c6bf0 2023-10-06T01:41:27.123 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.117+0000 7f1f772ca700 1 -- 172.21.15.27:0/4213697154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f70064050 con 0x7f1f700c6bf0 2023-10-06T01:41:27.123 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.118+0000 7f1f6d7fa700 1 -- 172.21.15.27:0/4213697154 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1f6801c620 con 0x7f1f700c6bf0 2023-10-06T01:41:27.123 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.118+0000 7f1f6d7fa700 1 --2- 172.21.15.27:0/4213697154 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1f5c066f30 0x7f1f5c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:27.123 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.118+0000 7f1f6d7fa700 1 -- 172.21.15.27:0/4213697154 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f1f68085f70 con 0x7f1f700c6bf0 2023-10-06T01:41:27.123 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.118+0000 7f1f74865700 1 --2- 172.21.15.27:0/4213697154 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1f5c066f30 0x7f1f5c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:27.123 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.119+0000 7f1f74865700 1 --2- 172.21.15.27:0/4213697154 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1f5c066f30 0x7f1f5c0693e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1f700d0490 tx=0x7f1f64008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:27.123 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.122+0000 7f1f6d7fa700 1 -- 172.21.15.27:0/4213697154 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1f68053060 con 0x7f1f700c6bf0 2023-10-06T01:41:27.317 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.316+0000 7f1f772ca700 1 -- 172.21.15.27:0/4213697154 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f70159780 con 0x7f1f700c6bf0 2023-10-06T01:41:27.318 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.317+0000 7f1f6d7fa700 1 -- 172.21.15.27:0/4213697154 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16866 (secure 0 0 0) 0x7f1f68056680 con 0x7f1f700c6bf0 2023-10-06T01:41:27.318 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:27.319 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":20,"fsid":"20a7bff0-74f8-4f2f-b6c6-899e804bc965","created":"2023-10-06T01:40:54.474215+0000","modified":"2023-10-06T01:41:14.324546+0000","last_up_change":"2023-10-06T01:41:05.242847+0000","last_in_change":"2023-10-06T01:40:56.890960+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-10-06T01:41:05.685386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-10-06T01:41:06.092995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T01:41:10.434257+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T01:41:11.700262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b902c86d-800c-40c5-b152-b545e50d02c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":90644},{"type":"v1","addr":"172.21.15.27:6801","nonce":90644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":90644},{"type":"v1","addr":"172.21.15.27:6803","nonce":90644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":90644},{"type":"v1","addr":"172.21.15.27:6807","nonce":90644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":90644},{"type":"v1","addr":"172.21.15.27:6805","nonce":90644}]},"public_addr":"172.21.15.27:6801/90644","cluster_addr":"172.21.15.27:6803/90644","heartbeat_back_addr":"172.21.15.27:6807/90644","heartbeat_front_addr":"172.21.15.27:6805/90644","state":["exists","up"]},{"osd":1,"uuid":"5b4a70e5-b06e-48cd-bfc4-7962c5d05386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":90643},{"type":"v1","addr":"172.21.15.27:6809","nonce":90643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":90643},{"type":"v1","addr":"172.21.15.27:6811","nonce":90643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":90643},{"type":"v1","addr":"172.21.15.27:6815","nonce":90643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":90643},{"type":"v1","addr":"172.21.15.27:6813","nonce":90643}]},"public_addr":"172.21.15.27:6809/90643","cluster_addr":"172.21.15.27:6811/90643","heartbeat_back_addr":"172.21.15.27:6815/90643","heartbeat_front_addr":"172.21.15.27:6813/90643","state":["exists","up"]},{"osd":2,"uuid":"22d9de46-4073-4336-9122-5227e601eca0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":90645},{"type":"v1","addr":"172.21.15.27:6817","nonce":90645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":90645},{"type":"v1","addr":"172.21.15.27:6819","nonce":90645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":90645},{"type":"v1","addr":"172.21.15.27:6823","nonce":90645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":90645},{"type":"v1","addr":"172.21.15.27:6821","nonce":90645}]},"public_addr":"172.21.15.27:6817/90645","cluster_addr":"172.21.15.27:6819/90645","heartbeat_back_addr":"172.21.15.27:6823/90645","heartbeat_front_addr":"172.21.15.27:6821/90645","state":["exists","up"]},{"osd":3,"uuid":"438bb2d3-34e2-4b7c-9734-8360c1b67650","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":90668},{"type":"v1","addr":"172.21.15.27:6825","nonce":90668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":90668},{"type":"v1","addr":"172.21.15.27:6827","nonce":90668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":90668},{"type":"v1","addr":"172.21.15.27:6831","nonce":90668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":90668},{"type":"v1","addr":"172.21.15.27:6829","nonce":90668}]},"public_addr":"172.21.15.27:6825/90668","cluster_addr":"172.21.15.27:6827/90668","heartbeat_back_addr":"172.21.15.27:6831/90668","heartbeat_front_addr":"172.21.15.27:6829/90668","state":["exists","up"]},{"osd":4,"uuid":"e1956a8e-e0f8-4e7c-acbb-40037415a354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":90863},{"type":"v1","addr":"172.21.15.124:6801","nonce":90863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":90863},{"type":"v1","addr":"172.21.15.124:6803","nonce":90863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":90863},{"type":"v1","addr":"172.21.15.124:6807","nonce":90863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":90863},{"type":"v1","addr":"172.21.15.124:6805","nonce":90863}]},"public_addr":"172.21.15.124:6801/90863","cluster_addr":"172.21.15.124:6803/90863","heartbeat_back_addr":"172.21.15.124:6807/90863","heartbeat_front_addr":"172.21.15.124:6805/90863","state":["exists","up"]},{"osd":5,"uuid":"306c59d4-846d-4b73-b1c9-051bec9e9368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":90885},{"type":"v1","addr":"172.21.15.124:6817","nonce":90885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":90885},{"type":"v1","addr":"172.21.15.124:6819","nonce":90885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":90885},{"type":"v1","addr":"172.21.15.124:6823","nonce":90885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":90885},{"type":"v1","addr":"172.21.15.124:6821","nonce":90885}]},"public_addr":"172.21.15.124:6817/90885","cluster_addr":"172.21.15.124:6819/90885","heartbeat_back_addr":"172.21.15.124:6823/90885","heartbeat_front_addr":"172.21.15.124:6821/90885","state":["exists","up"]},{"osd":6,"uuid":"15463e53-ac8d-47d1-b4f6-c0d31f29e4a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":90864},{"type":"v1","addr":"172.21.15.124:6809","nonce":90864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":90864},{"type":"v1","addr":"172.21.15.124:6811","nonce":90864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":90864},{"type":"v1","addr":"172.21.15.124:6815","nonce":90864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":90864},{"type":"v1","addr":"172.21.15.124:6813","nonce":90864}]},"public_addr":"172.21.15.124:6809/90864","cluster_addr":"172.21.15.124:6811/90864","heartbeat_back_addr":"172.21.15.124:6815/90864","heartbeat_front_addr":"172.21.15.124:6813/90864","state":["exists","up"]},{"osd":7,"uuid":"ae463a90-a9d3-498f-86fa-47a35b3fe889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":90894},{"type":"v1","addr":"172.21.15.124:6825","nonce":90894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":90894},{"type":"v1","addr":"172.21.15.124:6827","nonce":90894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":90894},{"type":"v1","addr":"172.21.15.124:6831","nonce":90894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":90894},{"type":"v1","addr":"172.21.15.124:6829","nonce":90894}]},"public_addr":"172.21.15.124:6825/90894","cluster_addr":"172.21.15.124:6827/90894","heartbeat_back_addr":"172.21.15.124:6831/90894","heartbeat_front_addr":"172.21.15.124:6829/90894","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.827704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.839951+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.032230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.050605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.987908+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.031290+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.080707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.053692+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T01:41:27.320 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.319+0000 7f1f772ca700 1 -- 172.21.15.27:0/4213697154 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1f5c066f30 msgr2=0x7f1f5c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:27.321 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.319+0000 7f1f772ca700 1 --2- 172.21.15.27:0/4213697154 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1f5c066f30 0x7f1f5c0693e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1f700d0490 tx=0x7f1f64008040 comp rx=0 tx=0).stop 2023-10-06T01:41:27.321 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.319+0000 7f1f772ca700 1 -- 172.21.15.27:0/4213697154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f700c6bf0 msgr2=0x7f1f701582f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:27.321 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.319+0000 7f1f772ca700 1 --2- 172.21.15.27:0/4213697154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f700c6bf0 0x7f1f701582f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1f68009260 tx=0x7f1f68016000 comp rx=0 tx=0).stop 2023-10-06T01:41:27.321 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.319+0000 7f1f772ca700 1 -- 172.21.15.27:0/4213697154 shutdown_connections 2023-10-06T01:41:27.321 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.319+0000 7f1f772ca700 1 --2- 172.21.15.27:0/4213697154 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1f5c066f30 0x7f1f5c0693e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:27.321 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.319+0000 7f1f772ca700 1 --2- 172.21.15.27:0/4213697154 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1f700d1030 0x7f1f700ceb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:27.321 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.319+0000 7f1f772ca700 1 --2- 172.21.15.27:0/4213697154 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1f700c6bf0 0x7f1f701582f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:27.321 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.319+0000 7f1f772ca700 1 --2- 172.21.15.27:0/4213697154 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1f700cdc00 0x7f1f700ce5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:27.321 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.319+0000 7f1f772ca700 1 -- 172.21.15.27:0/4213697154 >> 172.21.15.27:0/4213697154 conn(0x7f1f70099990 msgr2=0x7f1f700c77a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:27.321 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.319+0000 7f1f772ca700 1 -- 172.21.15.27:0/4213697154 shutdown_connections 2023-10-06T01:41:27.322 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.319+0000 7f1f772ca700 1 -- 172.21.15.27:0/4213697154 wait complete. 2023-10-06T01:41:27.333 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-10-06T01:41:05.685386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-10-06T01:41:06.092995+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-06T01:41:10.434257+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-10-06T01:41:11.700262+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-06T01:41:27.333 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-06T01:41:27.455 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.453+0000 7f5c767f6700 1 Processor -- start 2023-10-06T01:41:27.455 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.454+0000 7f5c767f6700 1 -- start start 2023-10-06T01:41:27.456 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.454+0000 7f5c767f6700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5c7013fb10 0x7f5c7006a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:27.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.454+0000 7f5c767f6700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5c7013fee0 0x7f5c700c79c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:27.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.454+0000 7f5c767f6700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c700c87a0 con 0x7f5c7006f270 2023-10-06T01:41:27.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.454+0000 7f5c767f6700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c7006aaf0 con 0x7f5c7013fee0 2023-10-06T01:41:27.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.454+0000 7f5c767f6700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7006adb0 con 0x7f5c7013fb10 2023-10-06T01:41:27.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.455+0000 7f5c6f7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5c7013fee0 0x7f5c700c79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:27.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.455+0000 7f5c74d93700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f5c7006f270 0x7f5c7015a000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57040/0 (socket says 172.21.15.27:57040) 2023-10-06T01:41:27.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.455+0000 7f5c74d93700 1 -- 172.21.15.27:0/3241931157 learned_addr learned my addr 172.21.15.27:0/3241931157 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:27.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.455+0000 7f5c6ffff700 1 --2- 172.21.15.27:0/3241931157 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5c7013fb10 0x7f5c7006a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:27.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.455+0000 7f5c6f7fe700 1 -- 172.21.15.27:0/3241931157 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5c7013fb10 msgr2=0x7f5c7006a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:27.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.455+0000 7f5c6f7fe700 1 --2- 172.21.15.27:0/3241931157 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5c7013fb10 0x7f5c7006a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:27.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.455+0000 7f5c6f7fe700 1 -- 172.21.15.27:0/3241931157 >> v1:172.21.15.27:6789/0 conn(0x7f5c7006f270 legacy=0x7f5c7015a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:27.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.455+0000 7f5c6f7fe700 1 -- 172.21.15.27:0/3241931157 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c7006b090 con 0x7f5c7013fee0 2023-10-06T01:41:27.458 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.455+0000 7f5c6effd700 1 -- 172.21.15.27:0/3241931157 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2369588810 0 0) 0x7f5c700c87a0 con 0x7f5c7006f270 2023-10-06T01:41:27.458 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.455+0000 7f5c6f7fe700 1 --2- 172.21.15.27:0/3241931157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c7013fee0 0x7f5c700c79c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5c640098e0 tx=0x7f5c640091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb13bfe0aee96db8 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:27.458 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.455+0000 7f5c6effd700 1 -- 172.21.15.27:0/3241931157 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6400b640 con 0x7f5c7013fee0 2023-10-06T01:41:27.458 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.455+0000 7f5c6effd700 1 -- 172.21.15.27:0/3241931157 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c6400b7a0 con 0x7f5c7013fee0 2023-10-06T01:41:27.458 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.455+0000 7f5c6effd700 1 -- 172.21.15.27:0/3241931157 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6400bb40 con 0x7f5c7013fee0 2023-10-06T01:41:27.461 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.456+0000 7f5c767f6700 1 -- 172.21.15.27:0/3241931157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c7013fee0 msgr2=0x7f5c700c79c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:27.461 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.456+0000 7f5c767f6700 1 --2- 172.21.15.27:0/3241931157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c7013fee0 0x7f5c700c79c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5c640098e0 tx=0x7f5c640091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:27.461 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.456+0000 7f5c767f6700 1 -- 172.21.15.27:0/3241931157 shutdown_connections 2023-10-06T01:41:27.461 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.456+0000 7f5c767f6700 1 --2- 172.21.15.27:0/3241931157 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5c7013fb10 0x7f5c7006a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:27.461 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.456+0000 7f5c767f6700 1 --2- 172.21.15.27:0/3241931157 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c7013fee0 0x7f5c700c79c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:27.461 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.456+0000 7f5c767f6700 1 -- 172.21.15.27:0/3241931157 >> 172.21.15.27:0/3241931157 conn(0x7f5c7006e760 msgr2=0x7f5c7006eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:27.461 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.456+0000 7f5c767f6700 1 -- 172.21.15.27:0/3241931157 shutdown_connections 2023-10-06T01:41:27.461 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.456+0000 7f5c767f6700 1 -- 172.21.15.27:0/3241931157 wait complete. 2023-10-06T01:41:27.462 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.457+0000 7f5c767f6700 1 Processor -- start 2023-10-06T01:41:27.462 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.457+0000 7f5c767f6700 1 -- start start 2023-10-06T01:41:27.462 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.458+0000 7f5c767f6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c7006f270 0x7f5c700c5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:27.462 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.458+0000 7f5c767f6700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5c7013fb10 0x7f5c700b1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:27.462 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.458+0000 7f5c767f6700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5c7013fee0 0x7f5c7015c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:27.462 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.458+0000 7f5c767f6700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c700b1750 con 0x7f5c7006f270 2023-10-06T01:41:27.462 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.458+0000 7f5c767f6700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c700c6240 con 0x7f5c7013fee0 2023-10-06T01:41:27.462 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.458+0000 7f5c767f6700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c700c6520 con 0x7f5c7013fb10 2023-10-06T01:41:27.462 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.458+0000 7f5c6ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c7006f270 0x7f5c700c5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:27.462 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.458+0000 7f5c6ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c7006f270 0x7f5c700c5b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52334/0 (socket says 172.21.15.27:52334) 2023-10-06T01:41:27.462 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.458+0000 7f5c6ffff700 1 -- 172.21.15.27:0/145288169 learned_addr learned my addr 172.21.15.27:0/145288169 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:27.462 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.458+0000 7f5c74d93700 1 --2- 172.21.15.27:0/145288169 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5c7013fee0 0x7f5c7015c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:27.463 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.458+0000 7f5c6f7fe700 1 --2- 172.21.15.27:0/145288169 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5c7013fb10 0x7f5c700b1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:27.463 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.458+0000 7f5c6ffff700 1 -- 172.21.15.27:0/145288169 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5c7013fb10 msgr2=0x7f5c700b1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:27.463 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.458+0000 7f5c6ffff700 1 --2- 172.21.15.27:0/145288169 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5c7013fb10 0x7f5c700b1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:27.463 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.458+0000 7f5c6ffff700 1 -- 172.21.15.27:0/145288169 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5c7013fee0 msgr2=0x7f5c7015c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:27.463 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.458+0000 7f5c6ffff700 1 --2- 172.21.15.27:0/145288169 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5c7013fee0 0x7f5c7015c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:27.463 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.458+0000 7f5c6ffff700 1 -- 172.21.15.27:0/145288169 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c64009420 con 0x7f5c7006f270 2023-10-06T01:41:27.463 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.458+0000 7f5c74d93700 1 --2- 172.21.15.27:0/145288169 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5c7013fee0 0x7f5c7015c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:41:27.463 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.458+0000 7f5c6f7fe700 1 --2- 172.21.15.27:0/145288169 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5c7013fb10 0x7f5c700b1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:41:27.463 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.458+0000 7f5c6ffff700 1 --2- 172.21.15.27:0/145288169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c7006f270 0x7f5c700c5b40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5c6000ec80 tx=0x7f5c600091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:27.463 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.459+0000 7f5c57fff700 1 -- 172.21.15.27:0/145288169 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6000c9d0 con 0x7f5c7006f270 2023-10-06T01:41:27.463 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.459+0000 7f5c57fff700 1 -- 172.21.15.27:0/145288169 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c6000cb30 con 0x7f5c7006f270 2023-10-06T01:41:27.463 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.459+0000 7f5c57fff700 1 -- 172.21.15.27:0/145288169 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c60013470 con 0x7f5c7006f270 2023-10-06T01:41:27.464 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.459+0000 7f5c767f6700 1 -- 172.21.15.27:0/145288169 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c700c6800 con 0x7f5c7006f270 2023-10-06T01:41:27.464 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.459+0000 7f5c767f6700 1 -- 172.21.15.27:0/145288169 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c700c6dc0 con 0x7f5c7006f270 2023-10-06T01:41:27.464 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.460+0000 7f5c767f6700 1 -- 172.21.15.27:0/145288169 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c70064050 con 0x7f5c7006f270 2023-10-06T01:41:27.464 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.460+0000 7f5c57fff700 1 -- 172.21.15.27:0/145288169 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5c60013820 con 0x7f5c7006f270 2023-10-06T01:41:27.464 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.461+0000 7f5c57fff700 1 --2- 172.21.15.27:0/145288169 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5c5c066f30 0x7f5c5c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:27.465 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.461+0000 7f5c57fff700 1 -- 172.21.15.27:0/145288169 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f5c60084a20 con 0x7f5c7006f270 2023-10-06T01:41:27.465 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.461+0000 7f5c6f7fe700 1 --2- 172.21.15.27:0/145288169 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5c5c066f30 0x7f5c5c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:27.465 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.461+0000 7f5c6f7fe700 1 --2- 172.21.15.27:0/145288169 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5c5c066f30 0x7f5c5c0693e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5c70166a40 tx=0x7f5c64017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:27.466 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.465+0000 7f5c57fff700 1 -- 172.21.15.27:0/145288169 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5c60051b30 con 0x7f5c7006f270 2023-10-06T01:41:27.659 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.658+0000 7f5c767f6700 1 -- 172.21.15.27:0/145288169 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f5c700cce00 con 0x7f5c7006f270 2023-10-06T01:41:27.659 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.658+0000 7f5c57fff700 1 -- 172.21.15.27:0/145288169 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f5c60055150 con 0x7f5c7006f270 2023-10-06T01:41:27.660 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 8 2023-10-06T01:41:27.662 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.660+0000 7f5c767f6700 1 -- 172.21.15.27:0/145288169 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5c5c066f30 msgr2=0x7f5c5c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:27.662 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.660+0000 7f5c767f6700 1 --2- 172.21.15.27:0/145288169 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5c5c066f30 0x7f5c5c0693e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5c70166a40 tx=0x7f5c64017000 comp rx=0 tx=0).stop 2023-10-06T01:41:27.662 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.660+0000 7f5c767f6700 1 -- 172.21.15.27:0/145288169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c7006f270 msgr2=0x7f5c700c5b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:27.662 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.660+0000 7f5c767f6700 1 --2- 172.21.15.27:0/145288169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c7006f270 0x7f5c700c5b40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5c6000ec80 tx=0x7f5c600091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:27.662 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.660+0000 7f5c767f6700 1 -- 172.21.15.27:0/145288169 shutdown_connections 2023-10-06T01:41:27.662 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.660+0000 7f5c767f6700 1 --2- 172.21.15.27:0/145288169 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5c5c066f30 0x7f5c5c0693e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:27.662 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.660+0000 7f5c767f6700 1 --2- 172.21.15.27:0/145288169 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5c7013fee0 0x7f5c7015c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:27.662 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.660+0000 7f5c767f6700 1 --2- 172.21.15.27:0/145288169 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c7006f270 0x7f5c700c5b40 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:27.662 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.660+0000 7f5c767f6700 1 --2- 172.21.15.27:0/145288169 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5c7013fb10 0x7f5c700b1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:27.662 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.660+0000 7f5c767f6700 1 -- 172.21.15.27:0/145288169 >> 172.21.15.27:0/145288169 conn(0x7f5c7006e760 msgr2=0x7f5c700cb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:27.663 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.660+0000 7f5c767f6700 1 -- 172.21.15.27:0/145288169 shutdown_connections 2023-10-06T01:41:27.663 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.660+0000 7f5c767f6700 1 -- 172.21.15.27:0/145288169 wait complete. 2023-10-06T01:41:27.673 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-06T01:41:27.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.794+0000 7f182baad700 1 Processor -- start 2023-10-06T01:41:27.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.794+0000 7f182baad700 1 -- start start 2023-10-06T01:41:27.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.795+0000 7f182baad700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f182415cb10 0x7f18240af780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:27.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.795+0000 7f182baad700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f18240b1690 0x7f18240b1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:27.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.795+0000 7f182baad700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18240698e0 con 0x7f18240b4000 2023-10-06T01:41:27.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.795+0000 7f182baad700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18240b2020 con 0x7f18240b1690 2023-10-06T01:41:27.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.795+0000 7f182baad700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f182415a260 con 0x7f182415cb10 2023-10-06T01:41:27.798 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.795+0000 7f1829849700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f182415cb10 0x7f18240af780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:27.798 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.795+0000 7f1829849700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f182415cb10 0x7f18240af780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.27:44768/0 (socket says 172.21.15.27:44768) 2023-10-06T01:41:27.798 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.795+0000 7f1829849700 1 -- 172.21.15.27:0/264782647 learned_addr learned my addr 172.21.15.27:0/264782647 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:27.798 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.795+0000 7f182a04a700 1 --2- 172.21.15.27:0/264782647 >> v2:172.21.15.124:3300/0 conn(0x7f18240b1690 0x7f18240b1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:27.798 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.795+0000 7f1829849700 1 -- 172.21.15.27:0/264782647 >> v2:172.21.15.124:3300/0 conn(0x7f18240b1690 msgr2=0x7f18240b1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:27.798 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.795+0000 7f1829849700 1 --2- 172.21.15.27:0/264782647 >> v2:172.21.15.124:3300/0 conn(0x7f18240b1690 0x7f18240b1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:27.798 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.795+0000 7f1829849700 1 -- 172.21.15.27:0/264782647 >> v1:172.21.15.124:6789/0 conn(0x7f18240b4000 legacy=0x7f1824159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:27.798 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.795+0000 7f1829849700 1 -- 172.21.15.27:0/264782647 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1824068fd0 con 0x7f182415cb10 2023-10-06T01:41:27.798 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.796+0000 7f1829849700 1 --2- 172.21.15.27:0/264782647 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f182415cb10 0x7f18240af780 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f181c0099b0 tx=0x7f181c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e874d3b6fa21641 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:27.798 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.796+0000 7f1828847700 1 -- 172.21.15.27:0/264782647 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181c009ea0 con 0x7f182415cb10 2023-10-06T01:41:27.798 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.796+0000 7f1828847700 1 -- 172.21.15.27:0/264782647 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f181c00b860 con 0x7f182415cb10 2023-10-06T01:41:27.798 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.796+0000 7f1828847700 1 -- 172.21.15.27:0/264782647 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f181c00bc00 con 0x7f182415cb10 2023-10-06T01:41:27.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.796+0000 7f182baad700 1 -- 172.21.15.27:0/264782647 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f182415cb10 msgr2=0x7f18240af780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:27.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.796+0000 7f182baad700 1 --2- 172.21.15.27:0/264782647 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f182415cb10 0x7f18240af780 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f181c0099b0 tx=0x7f181c009250 comp rx=0 tx=0).stop 2023-10-06T01:41:27.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.796+0000 7f182baad700 1 -- 172.21.15.27:0/264782647 shutdown_connections 2023-10-06T01:41:27.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.796+0000 7f182baad700 1 --2- 172.21.15.27:0/264782647 >> v2:172.21.15.124:3300/0 conn(0x7f18240b1690 0x7f18240b1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:27.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.796+0000 7f182baad700 1 --2- 172.21.15.27:0/264782647 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f182415cb10 0x7f18240af780 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:27.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.796+0000 7f182baad700 1 -- 172.21.15.27:0/264782647 >> 172.21.15.27:0/264782647 conn(0x7f182406d890 msgr2=0x7f182406dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:27.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.796+0000 7f182baad700 1 -- 172.21.15.27:0/264782647 shutdown_connections 2023-10-06T01:41:27.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.796+0000 7f182baad700 1 -- 172.21.15.27:0/264782647 wait complete. 2023-10-06T01:41:27.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.797+0000 7f182baad700 1 Processor -- start 2023-10-06T01:41:27.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.798+0000 7f182baad700 1 -- start start 2023-10-06T01:41:27.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.798+0000 7f182baad700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18240b1690 0x7f18241572f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:27.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.798+0000 7f182baad700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18240b4000 0x7f182416ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:27.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.798+0000 7f182baad700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f182415cb10 0x7f182415b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:27.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.798+0000 7f182baad700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182416b2a0 con 0x7f18240b4000 2023-10-06T01:41:27.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.798+0000 7f182baad700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18240cc7e0 con 0x7f18240b1690 2023-10-06T01:41:27.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.798+0000 7f182baad700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18240a6d40 con 0x7f182415cb10 2023-10-06T01:41:27.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.798+0000 7f1829048700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18240b4000 0x7f182416ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:27.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.798+0000 7f1829048700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18240b4000 0x7f182416ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52342/0 (socket says 172.21.15.27:52342) 2023-10-06T01:41:27.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.798+0000 7f1829048700 1 -- 172.21.15.27:0/1020526239 learned_addr learned my addr 172.21.15.27:0/1020526239 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:27.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.798+0000 7f182a04a700 1 --2- 172.21.15.27:0/1020526239 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f182415cb10 0x7f182415b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:27.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.798+0000 7f1829849700 1 --2- 172.21.15.27:0/1020526239 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18240b1690 0x7f18241572f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:27.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.799+0000 7f182a04a700 1 -- 172.21.15.27:0/1020526239 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18240b1690 msgr2=0x7f18241572f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:27.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.799+0000 7f182a04a700 1 --2- 172.21.15.27:0/1020526239 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18240b1690 0x7f18241572f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:27.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.799+0000 7f182a04a700 1 -- 172.21.15.27:0/1020526239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18240b4000 msgr2=0x7f182416ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:27.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.799+0000 7f182a04a700 1 --2- 172.21.15.27:0/1020526239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18240b4000 0x7f182416ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:27.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.799+0000 7f182a04a700 1 -- 172.21.15.27:0/1020526239 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f181c0094f0 con 0x7f182415cb10 2023-10-06T01:41:27.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.799+0000 7f1829048700 1 --2- 172.21.15.27:0/1020526239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18240b4000 0x7f182416ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:41:27.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.799+0000 7f182a04a700 1 --2- 172.21.15.27:0/1020526239 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f182415cb10 0x7f182415b230 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f181400e950 tx=0x7f18140091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:27.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.799+0000 7f1821ffb700 1 -- 172.21.15.27:0/1020526239 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1814013680 con 0x7f182415cb10 2023-10-06T01:41:27.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.799+0000 7f1821ffb700 1 -- 172.21.15.27:0/1020526239 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1814013800 con 0x7f182415cb10 2023-10-06T01:41:27.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.799+0000 7f1821ffb700 1 -- 172.21.15.27:0/1020526239 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1814013bc0 con 0x7f182415cb10 2023-10-06T01:41:27.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.799+0000 7f182baad700 1 -- 172.21.15.27:0/1020526239 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18240ccaa0 con 0x7f182415cb10 2023-10-06T01:41:27.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.799+0000 7f182baad700 1 -- 172.21.15.27:0/1020526239 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18240cd060 con 0x7f182415cb10 2023-10-06T01:41:27.804 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.800+0000 7f182baad700 1 -- 172.21.15.27:0/1020526239 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1824064050 con 0x7f182415cb10 2023-10-06T01:41:27.804 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.800+0000 7f1821ffb700 1 -- 172.21.15.27:0/1020526239 <== mon.2 v2:172.21.15.124:3301/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f181400c3e0 con 0x7f182415cb10 2023-10-06T01:41:27.804 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.801+0000 7f1821ffb700 1 --2- 172.21.15.27:0/1020526239 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1810066f30 0x7f18100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:27.805 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.801+0000 7f1821ffb700 1 -- 172.21.15.27:0/1020526239 <== mon.2 v2:172.21.15.124:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f1814084620 con 0x7f182415cb10 2023-10-06T01:41:27.805 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.801+0000 7f1829849700 1 --2- 172.21.15.27:0/1020526239 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1810066f30 0x7f18100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:27.805 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.802+0000 7f1829849700 1 --2- 172.21.15.27:0/1020526239 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1810066f30 0x7f18100693e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f181c0099b0 tx=0x7f181c017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:27.807 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.806+0000 7f1821ffb700 1 -- 172.21.15.27:0/1020526239 <== mon.2 v2:172.21.15.124:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f18140551c0 con 0x7f182415cb10 2023-10-06T01:41:27.998 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.996+0000 7f182baad700 1 -- 172.21.15.27:0/1020526239 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f18240b0020 con 0x7f182415cb10 2023-10-06T01:41:27.998 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.996+0000 7f1821ffb700 1 -- 172.21.15.27:0/1020526239 <== mon.2 v2:172.21.15.124:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1814054d50 con 0x7f182415cb10 2023-10-06T01:41:27.998 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2023-10-06T01:41:28.000 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.998+0000 7f182baad700 1 -- 172.21.15.27:0/1020526239 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1810066f30 msgr2=0x7f18100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:28.000 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.998+0000 7f182baad700 1 --2- 172.21.15.27:0/1020526239 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1810066f30 0x7f18100693e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f181c0099b0 tx=0x7f181c017000 comp rx=0 tx=0).stop 2023-10-06T01:41:28.000 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.998+0000 7f182baad700 1 -- 172.21.15.27:0/1020526239 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f182415cb10 msgr2=0x7f182415b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:28.000 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.998+0000 7f182baad700 1 --2- 172.21.15.27:0/1020526239 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f182415cb10 0x7f182415b230 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f181400e950 tx=0x7f18140091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:28.000 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.998+0000 7f182baad700 1 -- 172.21.15.27:0/1020526239 shutdown_connections 2023-10-06T01:41:28.000 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.998+0000 7f182baad700 1 --2- 172.21.15.27:0/1020526239 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1810066f30 0x7f18100693e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:28.001 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.998+0000 7f182baad700 1 --2- 172.21.15.27:0/1020526239 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f182415cb10 0x7f182415b230 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:28.001 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.998+0000 7f182baad700 1 --2- 172.21.15.27:0/1020526239 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18240b1690 0x7f18241572f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:28.001 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.998+0000 7f182baad700 1 --2- 172.21.15.27:0/1020526239 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f18240b4000 0x7f182416ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:28.001 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.998+0000 7f182baad700 1 -- 172.21.15.27:0/1020526239 >> 172.21.15.27:0/1020526239 conn(0x7f182406d890 msgr2=0x7f182415c070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:28.001 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.999+0000 7f182baad700 1 -- 172.21.15.27:0/1020526239 shutdown_connections 2023-10-06T01:41:28.001 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:27.999+0000 7f182baad700 1 -- 172.21.15.27:0/1020526239 wait complete. 2023-10-06T01:41:28.012 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-06T01:41:28.135 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.133+0000 7f3499194700 1 Processor -- start 2023-10-06T01:41:28.135 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.133+0000 7f3499194700 1 -- start start 2023-10-06T01:41:28.137 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.134+0000 7f3499194700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3494142e90 0x7f3494069810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:28.137 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.134+0000 7f3499194700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f349415d9a0 0x7f34940b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:28.137 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.134+0000 7f3499194700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3494069380 con 0x7f3494069d50 2023-10-06T01:41:28.138 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.134+0000 7f3499194700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3494068770 con 0x7f3494142e90 2023-10-06T01:41:28.138 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.134+0000 7f3499194700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3494068a50 con 0x7f349415d9a0 2023-10-06T01:41:28.138 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.134+0000 7f3493fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3494142e90 0x7f3494069810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:28.138 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.134+0000 7f3493fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f3494142e90 0x7f3494069810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52356/0 (socket says 172.21.15.27:52356) 2023-10-06T01:41:28.138 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.134+0000 7f3493fff700 1 -- 172.21.15.27:0/668582503 learned_addr learned my addr 172.21.15.27:0/668582503 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:28.138 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.134+0000 7f3498993700 1 --2- 172.21.15.27:0/668582503 >> v2:172.21.15.124:3300/0 conn(0x7f349415d9a0 0x7f34940b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:28.138 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.134+0000 7f3493fff700 1 -- 172.21.15.27:0/668582503 >> v2:172.21.15.124:3300/0 conn(0x7f349415d9a0 msgr2=0x7f34940b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:28.138 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.134+0000 7f3493fff700 1 --2- 172.21.15.27:0/668582503 >> v2:172.21.15.124:3300/0 conn(0x7f349415d9a0 0x7f34940b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:28.138 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.134+0000 7f3493fff700 1 -- 172.21.15.27:0/668582503 >> v1:172.21.15.124:6789/0 conn(0x7f3494069d50 legacy=0x7f349415d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:28.138 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.134+0000 7f3493fff700 1 -- 172.21.15.27:0/668582503 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3494068d30 con 0x7f3494142e90 2023-10-06T01:41:28.139 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.135+0000 7f3493fff700 1 --2- 172.21.15.27:0/668582503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3494142e90 0x7f3494069810 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f34940b00f0 tx=0x7f3484009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1c496e65c0f0a33 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:28.139 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.135+0000 7f34937fe700 1 -- 172.21.15.27:0/668582503 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3484009e70 con 0x7f3494142e90 2023-10-06T01:41:28.139 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.135+0000 7f34937fe700 1 -- 172.21.15.27:0/668582503 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f348400b860 con 0x7f3494142e90 2023-10-06T01:41:28.139 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.135+0000 7f34937fe700 1 -- 172.21.15.27:0/668582503 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f348400bc00 con 0x7f3494142e90 2023-10-06T01:41:28.139 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.135+0000 7f3499194700 1 -- 172.21.15.27:0/668582503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3494142e90 msgr2=0x7f3494069810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:28.139 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.135+0000 7f3499194700 1 --2- 172.21.15.27:0/668582503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3494142e90 0x7f3494069810 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f34940b00f0 tx=0x7f3484009250 comp rx=0 tx=0).stop 2023-10-06T01:41:28.139 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.135+0000 7f3499194700 1 -- 172.21.15.27:0/668582503 shutdown_connections 2023-10-06T01:41:28.139 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.135+0000 7f3499194700 1 --2- 172.21.15.27:0/668582503 >> v2:172.21.15.124:3300/0 conn(0x7f349415d9a0 0x7f34940b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:28.139 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.135+0000 7f3499194700 1 --2- 172.21.15.27:0/668582503 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3494142e90 0x7f3494069810 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:28.139 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.135+0000 7f3499194700 1 -- 172.21.15.27:0/668582503 >> 172.21.15.27:0/668582503 conn(0x7f34940ccfe0 msgr2=0x7f34940cd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:28.139 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.135+0000 7f3499194700 1 -- 172.21.15.27:0/668582503 shutdown_connections 2023-10-06T01:41:28.140 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.135+0000 7f3499194700 1 -- 172.21.15.27:0/668582503 wait complete. 2023-10-06T01:41:28.142 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.136+0000 7f3499194700 1 Processor -- start 2023-10-06T01:41:28.143 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.137+0000 7f3499194700 1 -- start start 2023-10-06T01:41:28.143 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.137+0000 7f3499194700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3494069d50 0x7f349416d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:28.143 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.137+0000 7f3499194700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3494142e90 0x7f349416da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:28.143 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.137+0000 7f3499194700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f349415d9a0 0x7f349413e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:28.143 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.137+0000 7f3499194700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34940c6360 con 0x7f3494069d50 2023-10-06T01:41:28.143 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.137+0000 7f3499194700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34940c6640 con 0x7f349415d9a0 2023-10-06T01:41:28.143 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.137+0000 7f3499194700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34940c6920 con 0x7f3494142e90 2023-10-06T01:41:28.143 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.137+0000 7f3493fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3494069d50 0x7f349416d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:28.143 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.137+0000 7f3493fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3494069d50 0x7f349416d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52358/0 (socket says 172.21.15.27:52358) 2023-10-06T01:41:28.143 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.137+0000 7f3493fff700 1 -- 172.21.15.27:0/2227420616 learned_addr learned my addr 172.21.15.27:0/2227420616 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:28.144 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.138+0000 7f3498993700 1 --2- 172.21.15.27:0/2227420616 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f349415d9a0 0x7f349413e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:28.144 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.138+0000 7f348bfff700 1 --2- 172.21.15.27:0/2227420616 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3494142e90 0x7f349416da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:28.144 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.138+0000 7f3493fff700 1 -- 172.21.15.27:0/2227420616 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3494142e90 msgr2=0x7f349416da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:28.144 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.138+0000 7f3493fff700 1 --2- 172.21.15.27:0/2227420616 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3494142e90 0x7f349416da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:28.144 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.138+0000 7f3493fff700 1 -- 172.21.15.27:0/2227420616 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f349415d9a0 msgr2=0x7f349413e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:28.144 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.138+0000 7f3493fff700 1 --2- 172.21.15.27:0/2227420616 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f349415d9a0 0x7f349413e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:28.144 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.138+0000 7f3493fff700 1 -- 172.21.15.27:0/2227420616 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34840094f0 con 0x7f3494069d50 2023-10-06T01:41:28.144 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.138+0000 7f3498993700 1 --2- 172.21.15.27:0/2227420616 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f349415d9a0 0x7f349413e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:28.144 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.138+0000 7f348bfff700 1 --2- 172.21.15.27:0/2227420616 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3494142e90 0x7f349416da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:28.144 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.138+0000 7f3493fff700 1 --2- 172.21.15.27:0/2227420616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3494069d50 0x7f349416d530 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3494088d50 tx=0x7f3484017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:28.145 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.138+0000 7f3490ff9700 1 -- 172.21.15.27:0/2227420616 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3484011420 con 0x7f3494069d50 2023-10-06T01:41:28.145 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.138+0000 7f3490ff9700 1 -- 172.21.15.27:0/2227420616 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3484011580 con 0x7f3494069d50 2023-10-06T01:41:28.145 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.138+0000 7f3499194700 1 -- 172.21.15.27:0/2227420616 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34940c6c00 con 0x7f3494069d50 2023-10-06T01:41:28.145 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.138+0000 7f3490ff9700 1 -- 172.21.15.27:0/2227420616 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3484011940 con 0x7f3494069d50 2023-10-06T01:41:28.145 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.138+0000 7f3499194700 1 -- 172.21.15.27:0/2227420616 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34940c5e80 con 0x7f3494069d50 2023-10-06T01:41:28.145 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.139+0000 7f3499194700 1 -- 172.21.15.27:0/2227420616 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34940632e0 con 0x7f3494069d50 2023-10-06T01:41:28.145 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.139+0000 7f3490ff9700 1 -- 172.21.15.27:0/2227420616 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f348402c3b0 con 0x7f3494069d50 2023-10-06T01:41:28.145 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.140+0000 7f3490ff9700 1 --2- 172.21.15.27:0/2227420616 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3474066f30 0x7f34740693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:28.145 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.140+0000 7f3490ff9700 1 -- 172.21.15.27:0/2227420616 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f3484085d00 con 0x7f3494069d50 2023-10-06T01:41:28.145 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.140+0000 7f348bfff700 1 --2- 172.21.15.27:0/2227420616 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3474066f30 0x7f34740693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:28.146 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.140+0000 7f348bfff700 1 --2- 172.21.15.27:0/2227420616 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3474066f30 0x7f34740693e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f347c007fd0 tx=0x7f347c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:28.146 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.144+0000 7f3490ff9700 1 -- 172.21.15.27:0/2227420616 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f34940632e0 con 0x7f3494069d50 2023-10-06T01:41:28.340 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.339+0000 7f3499194700 1 -- 172.21.15.27:0/2227420616 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f34940632e0 con 0x7f3494069d50 2023-10-06T01:41:28.341 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.339+0000 7f3490ff9700 1 -- 172.21.15.27:0/2227420616 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3484015020 con 0x7f3494069d50 2023-10-06T01:41:28.341 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2023-10-06T01:41:28.343 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.342+0000 7f3499194700 1 -- 172.21.15.27:0/2227420616 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3474066f30 msgr2=0x7f34740693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:28.343 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.342+0000 7f3499194700 1 --2- 172.21.15.27:0/2227420616 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3474066f30 0x7f34740693e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f347c007fd0 tx=0x7f347c009040 comp rx=0 tx=0).stop 2023-10-06T01:41:28.344 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.342+0000 7f3499194700 1 -- 172.21.15.27:0/2227420616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3494069d50 msgr2=0x7f349416d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:28.344 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.342+0000 7f3499194700 1 --2- 172.21.15.27:0/2227420616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3494069d50 0x7f349416d530 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3494088d50 tx=0x7f3484017000 comp rx=0 tx=0).stop 2023-10-06T01:41:28.344 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.342+0000 7f3499194700 1 -- 172.21.15.27:0/2227420616 shutdown_connections 2023-10-06T01:41:28.344 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.342+0000 7f3499194700 1 --2- 172.21.15.27:0/2227420616 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3474066f30 0x7f34740693e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:28.344 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.342+0000 7f3499194700 1 --2- 172.21.15.27:0/2227420616 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f349415d9a0 0x7f349413e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:28.344 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.342+0000 7f3499194700 1 --2- 172.21.15.27:0/2227420616 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3494069d50 0x7f349416d530 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:28.344 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.342+0000 7f3499194700 1 --2- 172.21.15.27:0/2227420616 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3494142e90 0x7f349416da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:28.344 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.342+0000 7f3499194700 1 -- 172.21.15.27:0/2227420616 >> 172.21.15.27:0/2227420616 conn(0x7f34940ccfe0 msgr2=0x7f34941407c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:28.344 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.342+0000 7f3499194700 1 -- 172.21.15.27:0/2227420616 shutdown_connections 2023-10-06T01:41:28.344 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.342+0000 7f3499194700 1 -- 172.21.15.27:0/2227420616 wait complete. 2023-10-06T01:41:28.355 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-06T01:41:28.478 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.476+0000 7f349d9c7700 1 Processor -- start 2023-10-06T01:41:28.478 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.477+0000 7f349d9c7700 1 -- start start 2023-10-06T01:41:28.481 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.477+0000 7f349d9c7700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f34981598d0 0x7f349815d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:28.481 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.477+0000 7f349d9c7700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f349815d8e0 0x7f34980cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:28.481 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.477+0000 7f349d9c7700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f349815dd20 con 0x7f3498159500 2023-10-06T01:41:28.481 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.477+0000 7f349d9c7700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34980b1110 con 0x7f34981598d0 2023-10-06T01:41:28.481 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.477+0000 7f349d9c7700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34980b13b0 con 0x7f349815d8e0 2023-10-06T01:41:28.481 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.478+0000 7f34977fe700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f349815d8e0 0x7f34980cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:28.481 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.478+0000 7f34977fe700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f349815d8e0 0x7f34980cf6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.27:44798/0 (socket says 172.21.15.27:44798) 2023-10-06T01:41:28.481 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.478+0000 7f34977fe700 1 -- 172.21.15.27:0/2291399673 learned_addr learned my addr 172.21.15.27:0/2291399673 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:28.481 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.478+0000 7f34967fc700 1 --2- 172.21.15.27:0/2291399673 >> v2:172.21.15.124:3300/0 conn(0x7f34981598d0 0x7f349815d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:28.481 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.478+0000 7f34977fe700 1 -- 172.21.15.27:0/2291399673 >> v2:172.21.15.124:3300/0 conn(0x7f34981598d0 msgr2=0x7f349815d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.478+0000 7f34977fe700 1 --2- 172.21.15.27:0/2291399673 >> v2:172.21.15.124:3300/0 conn(0x7f34981598d0 0x7f349815d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.478+0000 7f34977fe700 1 -- 172.21.15.27:0/2291399673 >> v1:172.21.15.124:6789/0 conn(0x7f3498159500 legacy=0x7f34980b3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.478+0000 7f34977fe700 1 -- 172.21.15.27:0/2291399673 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34980b1690 con 0x7f349815d8e0 2023-10-06T01:41:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.478+0000 7f34967fc700 1 --2- 172.21.15.27:0/2291399673 >> v2:172.21.15.124:3300/0 conn(0x7f34981598d0 0x7f349815d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:41:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.478+0000 7f34977fe700 1 --2- 172.21.15.27:0/2291399673 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f349815d8e0 0x7f34980cf6c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3488009f20 tx=0x7f3488009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66ca5e8dc4e36f15 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.478+0000 7f3495ffb700 1 -- 172.21.15.27:0/2291399673 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f348800e070 con 0x7f349815d8e0 2023-10-06T01:41:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.478+0000 7f3495ffb700 1 -- 172.21.15.27:0/2291399673 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f348800b9d0 con 0x7f349815d8e0 2023-10-06T01:41:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.478+0000 7f3495ffb700 1 -- 172.21.15.27:0/2291399673 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34880092e0 con 0x7f349815d8e0 2023-10-06T01:41:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.479+0000 7f349d9c7700 1 -- 172.21.15.27:0/2291399673 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f349815d8e0 msgr2=0x7f34980cf6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.479+0000 7f349d9c7700 1 --2- 172.21.15.27:0/2291399673 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f349815d8e0 0x7f34980cf6c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3488009f20 tx=0x7f3488009840 comp rx=0 tx=0).stop 2023-10-06T01:41:28.482 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.479+0000 7f349d9c7700 1 -- 172.21.15.27:0/2291399673 shutdown_connections 2023-10-06T01:41:28.483 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.479+0000 7f349d9c7700 1 --2- 172.21.15.27:0/2291399673 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f349815d8e0 0x7f34980cf6c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:28.483 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.479+0000 7f349d9c7700 1 --2- 172.21.15.27:0/2291399673 >> v2:172.21.15.124:3300/0 conn(0x7f34981598d0 0x7f349815d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:28.483 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.479+0000 7f349d9c7700 1 -- 172.21.15.27:0/2291399673 >> 172.21.15.27:0/2291399673 conn(0x7f3498067630 msgr2=0x7f3498069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:28.483 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.479+0000 7f349d9c7700 1 -- 172.21.15.27:0/2291399673 shutdown_connections 2023-10-06T01:41:28.483 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.479+0000 7f349d9c7700 1 -- 172.21.15.27:0/2291399673 wait complete. 2023-10-06T01:41:28.485 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.480+0000 7f349d9c7700 1 Processor -- start 2023-10-06T01:41:28.485 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.480+0000 7f349d9c7700 1 -- start start 2023-10-06T01:41:28.485 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.481+0000 7f349d9c7700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3498159500 0x7f349814e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:28.485 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.481+0000 7f349d9c7700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f34981598d0 0x7f3498141040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:28.486 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.481+0000 7f349d9c7700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f349815d8e0 0x7f349814cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:28.486 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.481+0000 7f349d9c7700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349814e040 con 0x7f3498159500 2023-10-06T01:41:28.486 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.481+0000 7f349d9c7700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349814d430 con 0x7f349815d8e0 2023-10-06T01:41:28.486 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.481+0000 7f349d9c7700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f349814d710 con 0x7f34981598d0 2023-10-06T01:41:28.486 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.481+0000 7f3496ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3498159500 0x7f349814e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:28.486 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.481+0000 7f3496ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3498159500 0x7f349814e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52368/0 (socket says 172.21.15.27:52368) 2023-10-06T01:41:28.486 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.481+0000 7f3496ffd700 1 -- 172.21.15.27:0/4140225673 learned_addr learned my addr 172.21.15.27:0/4140225673 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:28.486 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.481+0000 7f34967fc700 1 --2- 172.21.15.27:0/4140225673 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f34981598d0 0x7f3498141040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:28.486 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.481+0000 7f34977fe700 1 --2- 172.21.15.27:0/4140225673 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f349815d8e0 0x7f349814cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:28.486 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.481+0000 7f3496ffd700 1 -- 172.21.15.27:0/4140225673 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f34981598d0 msgr2=0x7f3498141040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:28.486 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.481+0000 7f3496ffd700 1 --2- 172.21.15.27:0/4140225673 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f34981598d0 0x7f3498141040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:28.487 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.481+0000 7f3496ffd700 1 -- 172.21.15.27:0/4140225673 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f349815d8e0 msgr2=0x7f349814cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:28.487 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.481+0000 7f3496ffd700 1 --2- 172.21.15.27:0/4140225673 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f349815d8e0 0x7f349814cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:28.487 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.481+0000 7f3496ffd700 1 -- 172.21.15.27:0/4140225673 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3480009420 con 0x7f3498159500 2023-10-06T01:41:28.487 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.482+0000 7f3496ffd700 1 --2- 172.21.15.27:0/4140225673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3498159500 0x7f349814e4d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f349000c590 tx=0x7f349000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:28.487 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.482+0000 7f348effd700 1 -- 172.21.15.27:0/4140225673 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3490011590 con 0x7f3498159500 2023-10-06T01:41:28.487 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.482+0000 7f348effd700 1 -- 172.21.15.27:0/4140225673 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3490011710 con 0x7f3498159500 2023-10-06T01:41:28.487 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.482+0000 7f349d9c7700 1 -- 172.21.15.27:0/4140225673 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3488009ae0 con 0x7f3498159500 2023-10-06T01:41:28.487 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.482+0000 7f348effd700 1 -- 172.21.15.27:0/4140225673 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3490011ad0 con 0x7f3498159500 2023-10-06T01:41:28.487 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.482+0000 7f349d9c7700 1 -- 172.21.15.27:0/4140225673 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f349814dcd0 con 0x7f3498159500 2023-10-06T01:41:28.487 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.483+0000 7f349d9c7700 1 -- 172.21.15.27:0/4140225673 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f349814e320 con 0x7f3498159500 2023-10-06T01:41:28.487 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.483+0000 7f348effd700 1 -- 172.21.15.27:0/4140225673 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f349001b3b0 con 0x7f3498159500 2023-10-06T01:41:28.488 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.483+0000 7f348effd700 1 --2- 172.21.15.27:0/4140225673 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3478066f30 0x7f34780693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:28.488 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.484+0000 7f348effd700 1 -- 172.21.15.27:0/4140225673 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f3490083bc0 con 0x7f3498159500 2023-10-06T01:41:28.488 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.484+0000 7f34967fc700 1 --2- 172.21.15.27:0/4140225673 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3478066f30 0x7f34780693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:28.488 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.484+0000 7f34967fc700 1 --2- 172.21.15.27:0/4140225673 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3478066f30 0x7f34780693e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3498153010 tx=0x7f34800091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:28.489 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.488+0000 7f348effd700 1 -- 172.21.15.27:0/4140225673 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3490050cd0 con 0x7f3498159500 2023-10-06T01:41:28.685 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.684+0000 7f349d9c7700 1 -- 172.21.15.27:0/4140225673 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3498064050 con 0x7f3498159500 2023-10-06T01:41:28.685 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.684+0000 7f348effd700 1 -- 172.21.15.27:0/4140225673 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3490017070 con 0x7f3498159500 2023-10-06T01:41:28.686 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2023-10-06T01:41:28.687 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.686+0000 7f349d9c7700 1 -- 172.21.15.27:0/4140225673 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3478066f30 msgr2=0x7f34780693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:28.688 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.686+0000 7f349d9c7700 1 --2- 172.21.15.27:0/4140225673 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3478066f30 0x7f34780693e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3498153010 tx=0x7f34800091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:28.688 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.686+0000 7f349d9c7700 1 -- 172.21.15.27:0/4140225673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3498159500 msgr2=0x7f349814e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:28.688 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.686+0000 7f349d9c7700 1 --2- 172.21.15.27:0/4140225673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3498159500 0x7f349814e4d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f349000c590 tx=0x7f349000c8a0 comp rx=0 tx=0).stop 2023-10-06T01:41:28.688 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.686+0000 7f349d9c7700 1 -- 172.21.15.27:0/4140225673 shutdown_connections 2023-10-06T01:41:28.688 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.686+0000 7f349d9c7700 1 --2- 172.21.15.27:0/4140225673 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3478066f30 0x7f34780693e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:28.688 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.686+0000 7f349d9c7700 1 --2- 172.21.15.27:0/4140225673 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f349815d8e0 0x7f349814cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:28.688 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.686+0000 7f349d9c7700 1 --2- 172.21.15.27:0/4140225673 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3498159500 0x7f349814e4d0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:28.688 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.686+0000 7f349d9c7700 1 --2- 172.21.15.27:0/4140225673 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f34981598d0 0x7f3498141040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:28.688 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.686+0000 7f349d9c7700 1 -- 172.21.15.27:0/4140225673 >> 172.21.15.27:0/4140225673 conn(0x7f3498067630 msgr2=0x7f34980b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:28.688 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.686+0000 7f349d9c7700 1 -- 172.21.15.27:0/4140225673 shutdown_connections 2023-10-06T01:41:28.688 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.686+0000 7f349d9c7700 1 -- 172.21.15.27:0/4140225673 wait complete. 2023-10-06T01:41:28.699 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T01:41:28.817 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.816+0000 7f232b53d700 1 Processor -- start 2023-10-06T01:41:28.818 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.816+0000 7f232b53d700 1 -- start start 2023-10-06T01:41:28.821 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.817+0000 7f232b53d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f23240cdc00 0x7f23240cbe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:28.821 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.817+0000 7f232b53d700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f23240c6dc0 0x7f23240ae5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:28.821 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.817+0000 7f232b53d700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23240ae140 con 0x7f23240d1030 2023-10-06T01:41:28.821 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.817+0000 7f232b53d700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23240cc390 con 0x7f23240c6dc0 2023-10-06T01:41:28.821 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.817+0000 7f232b53d700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23240ad530 con 0x7f23240cdc00 2023-10-06T01:41:28.822 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.817+0000 7f23292d9700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f23240cdc00 0x7f23240cbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:28.822 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.817+0000 7f2329ada700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f23240c6dc0 0x7f23240ae5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:28.822 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.817+0000 7f2328ad8700 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f23240d1030 0x7f23240c66b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.27:50932/0 (socket says 172.21.15.27:50932) 2023-10-06T01:41:28.822 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.817+0000 7f23292d9700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f23240cdc00 0x7f23240cbe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.27:44818/0 (socket says 172.21.15.27:44818) 2023-10-06T01:41:28.822 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.817+0000 7f2329ada700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f23240c6dc0 0x7f23240ae5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.27:37582/0 (socket says 172.21.15.27:37582) 2023-10-06T01:41:28.822 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.818+0000 7f2328ad8700 1 -- 172.21.15.27:0/2196728510 learned_addr learned my addr 172.21.15.27:0/2196728510 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:28.822 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.818+0000 7f23292d9700 1 -- 172.21.15.27:0/2196728510 >> v2:172.21.15.124:3300/0 conn(0x7f23240c6dc0 msgr2=0x7f23240ae5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:28.822 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.818+0000 7f23292d9700 1 --2- 172.21.15.27:0/2196728510 >> v2:172.21.15.124:3300/0 conn(0x7f23240c6dc0 0x7f23240ae5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:28.822 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.818+0000 7f23292d9700 1 -- 172.21.15.27:0/2196728510 >> v1:172.21.15.124:6789/0 conn(0x7f23240d1030 legacy=0x7f23240c66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:28.823 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.818+0000 7f23292d9700 1 -- 172.21.15.27:0/2196728510 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23240ad810 con 0x7f23240cdc00 2023-10-06T01:41:28.823 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.818+0000 7f23292d9700 1 --2- 172.21.15.27:0/2196728510 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f23240cdc00 0x7f23240cbe50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f231c009900 tx=0x7f231c009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=103472f385fc2729 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:28.823 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.818+0000 7f2323fff700 1 -- 172.21.15.27:0/2196728510 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231c005c10 con 0x7f23240cdc00 2023-10-06T01:41:28.823 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.818+0000 7f2323fff700 1 -- 172.21.15.27:0/2196728510 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f231c005d70 con 0x7f23240cdc00 2023-10-06T01:41:28.823 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.818+0000 7f2323fff700 1 -- 172.21.15.27:0/2196728510 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231c006260 con 0x7f23240cdc00 2023-10-06T01:41:28.823 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.819+0000 7f232b53d700 1 -- 172.21.15.27:0/2196728510 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f23240cdc00 msgr2=0x7f23240cbe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:28.823 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.819+0000 7f232b53d700 1 --2- 172.21.15.27:0/2196728510 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f23240cdc00 0x7f23240cbe50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f231c009900 tx=0x7f231c009120 comp rx=0 tx=0).stop 2023-10-06T01:41:28.823 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.819+0000 7f232b53d700 1 -- 172.21.15.27:0/2196728510 shutdown_connections 2023-10-06T01:41:28.823 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.819+0000 7f232b53d700 1 --2- 172.21.15.27:0/2196728510 >> v2:172.21.15.124:3300/0 conn(0x7f23240c6dc0 0x7f23240ae5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:28.823 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.819+0000 7f232b53d700 1 --2- 172.21.15.27:0/2196728510 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f23240cdc00 0x7f23240cbe50 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:28.823 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.819+0000 7f232b53d700 1 -- 172.21.15.27:0/2196728510 >> 172.21.15.27:0/2196728510 conn(0x7f2324099990 msgr2=0x7f23240b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:28.824 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.819+0000 7f232b53d700 1 -- 172.21.15.27:0/2196728510 shutdown_connections 2023-10-06T01:41:28.824 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.819+0000 7f232b53d700 1 -- 172.21.15.27:0/2196728510 wait complete. 2023-10-06T01:41:28.824 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.820+0000 7f232b53d700 1 Processor -- start 2023-10-06T01:41:28.824 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.820+0000 7f232b53d700 1 -- start start 2023-10-06T01:41:28.826 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.821+0000 7f232b53d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23240c6dc0 0x7f2324069d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:28.827 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.821+0000 7f232b53d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f23240cdc00 0x7f2324145ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:28.827 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.821+0000 7f232b53d700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f23240d1030 0x7f2324146430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:28.827 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.821+0000 7f232b53d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2324145c70 con 0x7f23240c6dc0 2023-10-06T01:41:28.827 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.821+0000 7f232b53d700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2324146970 con 0x7f23240d1030 2023-10-06T01:41:28.827 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.821+0000 7f232b53d700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f232409bbd0 con 0x7f23240cdc00 2023-10-06T01:41:28.827 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.821+0000 7f23292d9700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23240c6dc0 0x7f2324069d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:28.827 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.821+0000 7f23292d9700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23240c6dc0 0x7f2324069d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52370/0 (socket says 172.21.15.27:52370) 2023-10-06T01:41:28.827 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.821+0000 7f23292d9700 1 -- 172.21.15.27:0/133905541 learned_addr learned my addr 172.21.15.27:0/133905541 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:28.827 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.821+0000 7f2328ad8700 1 --2- 172.21.15.27:0/133905541 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f23240cdc00 0x7f2324145ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:28.827 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.821+0000 7f2329ada700 1 --2- 172.21.15.27:0/133905541 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f23240d1030 0x7f2324146430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:28.827 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.821+0000 7f23292d9700 1 -- 172.21.15.27:0/133905541 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f23240cdc00 msgr2=0x7f2324145ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:28.828 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.821+0000 7f23292d9700 1 --2- 172.21.15.27:0/133905541 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f23240cdc00 0x7f2324145ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:28.828 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.821+0000 7f23292d9700 1 -- 172.21.15.27:0/133905541 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f23240d1030 msgr2=0x7f2324146430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:28.828 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.821+0000 7f23292d9700 1 --2- 172.21.15.27:0/133905541 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f23240d1030 0x7f2324146430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:28.828 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.821+0000 7f23292d9700 1 -- 172.21.15.27:0/133905541 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f231c0093c0 con 0x7f23240c6dc0 2023-10-06T01:41:28.828 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.821+0000 7f2329ada700 1 --2- 172.21.15.27:0/133905541 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f23240d1030 0x7f2324146430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:41:28.828 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.821+0000 7f2328ad8700 1 --2- 172.21.15.27:0/133905541 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f23240cdc00 0x7f2324145ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:28.828 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.821+0000 7f23292d9700 1 --2- 172.21.15.27:0/133905541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23240c6dc0 0x7f2324069d40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f23240ad670 tx=0x7f231c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:28.828 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.822+0000 7f23217fa700 1 -- 172.21.15.27:0/133905541 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231c01d990 con 0x7f23240c6dc0 2023-10-06T01:41:28.828 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.822+0000 7f23217fa700 1 -- 172.21.15.27:0/133905541 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f231c01daf0 con 0x7f23240c6dc0 2023-10-06T01:41:28.828 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.822+0000 7f23217fa700 1 -- 172.21.15.27:0/133905541 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231c01b480 con 0x7f23240c6dc0 2023-10-06T01:41:28.828 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.822+0000 7f232b53d700 1 -- 172.21.15.27:0/133905541 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f232406a280 con 0x7f23240c6dc0 2023-10-06T01:41:28.828 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.822+0000 7f232b53d700 1 -- 172.21.15.27:0/133905541 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f232409b2a0 con 0x7f23240c6dc0 2023-10-06T01:41:28.829 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.822+0000 7f232b53d700 1 -- 172.21.15.27:0/133905541 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2324064050 con 0x7f23240c6dc0 2023-10-06T01:41:28.829 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.823+0000 7f23217fa700 1 -- 172.21.15.27:0/133905541 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f231c01b870 con 0x7f23240c6dc0 2023-10-06T01:41:28.829 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.823+0000 7f23217fa700 1 --2- 172.21.15.27:0/133905541 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2310066f30 0x7f23100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:28.829 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.823+0000 7f23217fa700 1 -- 172.21.15.27:0/133905541 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f231c0864a0 con 0x7f23240c6dc0 2023-10-06T01:41:28.829 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.824+0000 7f2328ad8700 1 --2- 172.21.15.27:0/133905541 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2310066f30 0x7f23100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:28.829 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.824+0000 7f2328ad8700 1 --2- 172.21.15.27:0/133905541 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2310066f30 0x7f23100693e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2318001820 tx=0x7f2318009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:28.829 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:28.828+0000 7f23217fa700 1 -- 172.21.15.27:0/133905541 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f231c053510 con 0x7f23240c6dc0 2023-10-06T01:41:29.064 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.063+0000 7f232b53d700 1 -- 172.21.15.27:0/133905541 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23240c60e0 con 0x7f23240c6dc0 2023-10-06T01:41:29.065 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.063+0000 7f23217fa700 1 -- 172.21.15.27:0/133905541 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4594 (secure 0 0 0) 0x7f231c056b30 con 0x7f23240c6dc0 2023-10-06T01:41:29.065 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:29.065 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4261,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6837/2699906167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6836","nonce":2699906167},{"type":"v1","addr":"172.21.15.124:6837","nonce":2699906167}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/313372271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":313372271},{"type":"v1","addr":"172.21.15.27:6834","nonce":313372271}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4287,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/4257743842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4257743842},{"type":"v1","addr":"172.21.15.27:6835","nonce":4257743842}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T01:41:12.735970+0000","modified":"2023-10-06T01:41:16.358924+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.124:6835/3803994994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":3803994994},{"type":"v1","addr":"172.21.15.124:6835","nonce":3803994994}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-06T01:41:29.067 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.065+0000 7f232b53d700 1 -- 172.21.15.27:0/133905541 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2310066f30 msgr2=0x7f23100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:29.067 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.065+0000 7f232b53d700 1 --2- 172.21.15.27:0/133905541 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2310066f30 0x7f23100693e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2318001820 tx=0x7f2318009040 comp rx=0 tx=0).stop 2023-10-06T01:41:29.068 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.065+0000 7f232b53d700 1 -- 172.21.15.27:0/133905541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23240c6dc0 msgr2=0x7f2324069d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:29.068 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.065+0000 7f232b53d700 1 --2- 172.21.15.27:0/133905541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23240c6dc0 0x7f2324069d40 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f23240ad670 tx=0x7f231c016000 comp rx=0 tx=0).stop 2023-10-06T01:41:29.068 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.065+0000 7f232b53d700 1 -- 172.21.15.27:0/133905541 shutdown_connections 2023-10-06T01:41:29.068 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.065+0000 7f232b53d700 1 --2- 172.21.15.27:0/133905541 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2310066f30 0x7f23100693e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:29.068 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.065+0000 7f232b53d700 1 --2- 172.21.15.27:0/133905541 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f23240d1030 0x7f2324146430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:29.069 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.065+0000 7f232b53d700 1 --2- 172.21.15.27:0/133905541 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f23240c6dc0 0x7f2324069d40 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:29.069 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.065+0000 7f232b53d700 1 --2- 172.21.15.27:0/133905541 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f23240cdc00 0x7f2324145ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:29.069 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.065+0000 7f232b53d700 1 -- 172.21.15.27:0/133905541 >> 172.21.15.27:0/133905541 conn(0x7f2324099990 msgr2=0x7f23240b2840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:29.069 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.065+0000 7f232b53d700 1 -- 172.21.15.27:0/133905541 shutdown_connections 2023-10-06T01:41:29.069 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.065+0000 7f232b53d700 1 -- 172.21.15.27:0/133905541 wait complete. 2023-10-06T01:41:29.069 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 7 2023-10-06T01:41:29.079 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T01:41:29.197 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.196+0000 7f0b4e2bc700 1 Processor -- start 2023-10-06T01:41:29.198 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.196+0000 7f0b4e2bc700 1 -- start start 2023-10-06T01:41:29.200 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.197+0000 7f0b4e2bc700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f0b480cdc00 0x7f0b480cbe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:29.200 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.197+0000 7f0b4e2bc700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0b480d1030 0x7f0b480c66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:29.200 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.197+0000 7f0b4e2bc700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b480ae140 con 0x7f0b480c6bf0 2023-10-06T01:41:29.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.197+0000 7f0b4e2bc700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b480cc390 con 0x7f0b480cdc00 2023-10-06T01:41:29.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.197+0000 7f0b4e2bc700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b480c7070 con 0x7f0b480d1030 2023-10-06T01:41:29.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.197+0000 7f0b4c859700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0b480c6bf0 0x7f0b480ae5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57048/0 (socket says 172.21.15.27:57048) 2023-10-06T01:41:29.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.197+0000 7f0b4c859700 1 -- 172.21.15.27:0/1896678768 learned_addr learned my addr 172.21.15.27:0/1896678768 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:29.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.197+0000 7f0b47fff700 1 --2- 172.21.15.27:0/1896678768 >> v2:172.21.15.124:3300/0 conn(0x7f0b480cdc00 0x7f0b480cbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:29.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.197+0000 7f0b477fe700 1 --2- 172.21.15.27:0/1896678768 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0b480d1030 0x7f0b480c66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:29.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.197+0000 7f0b46ffd700 1 -- 172.21.15.27:0/1896678768 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1324946406 0 0) 0x7f0b480ae140 con 0x7f0b480c6bf0 2023-10-06T01:41:29.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.198+0000 7f0b46ffd700 1 -- 172.21.15.27:0/1896678768 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b34003570 con 0x7f0b480c6bf0 2023-10-06T01:41:29.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.198+0000 7f0b47fff700 1 -- 172.21.15.27:0/1896678768 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0b480d1030 msgr2=0x7f0b480c66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:29.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.198+0000 7f0b47fff700 1 --2- 172.21.15.27:0/1896678768 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0b480d1030 0x7f0b480c66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.198+0000 7f0b47fff700 1 -- 172.21.15.27:0/1896678768 >> v1:172.21.15.27:6789/0 conn(0x7f0b480c6bf0 legacy=0x7f0b480ae5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.198+0000 7f0b47fff700 1 -- 172.21.15.27:0/1896678768 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b480ad530 con 0x7f0b480cdc00 2023-10-06T01:41:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.198+0000 7f0b477fe700 1 --2- 172.21.15.27:0/1896678768 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0b480d1030 0x7f0b480c66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:41:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.198+0000 7f0b47fff700 1 --2- 172.21.15.27:0/1896678768 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0b480cdc00 0x7f0b480cbe50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0b38009900 tx=0x7f0b38009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e70155ac753dc874 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.198+0000 7f0b46ffd700 1 -- 172.21.15.27:0/1896678768 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b38005c10 con 0x7f0b480cdc00 2023-10-06T01:41:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.198+0000 7f0b46ffd700 1 -- 172.21.15.27:0/1896678768 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b38005d70 con 0x7f0b480cdc00 2023-10-06T01:41:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.198+0000 7f0b46ffd700 1 -- 172.21.15.27:0/1896678768 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b38006240 con 0x7f0b480cdc00 2023-10-06T01:41:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.199+0000 7f0b4e2bc700 1 -- 172.21.15.27:0/1896678768 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0b480cdc00 msgr2=0x7f0b480cbe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.199+0000 7f0b4e2bc700 1 --2- 172.21.15.27:0/1896678768 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0b480cdc00 0x7f0b480cbe50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0b38009900 tx=0x7f0b38009120 comp rx=0 tx=0).stop 2023-10-06T01:41:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.199+0000 7f0b4e2bc700 1 -- 172.21.15.27:0/1896678768 shutdown_connections 2023-10-06T01:41:29.202 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.199+0000 7f0b4e2bc700 1 --2- 172.21.15.27:0/1896678768 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0b480cdc00 0x7f0b480cbe50 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:29.203 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.199+0000 7f0b4e2bc700 1 --2- 172.21.15.27:0/1896678768 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0b480d1030 0x7f0b480c66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:29.203 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.199+0000 7f0b4e2bc700 1 -- 172.21.15.27:0/1896678768 >> 172.21.15.27:0/1896678768 conn(0x7f0b48099990 msgr2=0x7f0b480b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:29.203 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.199+0000 7f0b4e2bc700 1 -- 172.21.15.27:0/1896678768 shutdown_connections 2023-10-06T01:41:29.203 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.199+0000 7f0b4e2bc700 1 -- 172.21.15.27:0/1896678768 wait complete. 2023-10-06T01:41:29.206 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.200+0000 7f0b4e2bc700 1 Processor -- start 2023-10-06T01:41:29.206 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.200+0000 7f0b4e2bc700 1 -- start start 2023-10-06T01:41:29.206 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.201+0000 7f0b4e2bc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b480c6bf0 0x7f0b48068f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:29.207 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.201+0000 7f0b4e2bc700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0b480cdc00 0x7f0b48169a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:29.207 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.201+0000 7f0b4e2bc700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0b480d1030 0x7f0b48157ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:29.207 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.201+0000 7f0b4e2bc700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4816a9f0 con 0x7f0b480c6bf0 2023-10-06T01:41:29.207 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.201+0000 7f0b4e2bc700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b48068aa0 con 0x7f0b480cdc00 2023-10-06T01:41:29.207 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.201+0000 7f0b4e2bc700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4816a430 con 0x7f0b480d1030 2023-10-06T01:41:29.208 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.201+0000 7f0b47fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b480c6bf0 0x7f0b48068f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:29.208 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.201+0000 7f0b47fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b480c6bf0 0x7f0b48068f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52376/0 (socket says 172.21.15.27:52376) 2023-10-06T01:41:29.208 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.201+0000 7f0b47fff700 1 -- 172.21.15.27:0/83892428 learned_addr learned my addr 172.21.15.27:0/83892428 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:29.208 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.201+0000 7f0b477fe700 1 --2- 172.21.15.27:0/83892428 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0b480cdc00 0x7f0b48169a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:29.208 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.201+0000 7f0b4c859700 1 --2- 172.21.15.27:0/83892428 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0b480d1030 0x7f0b48157ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:29.208 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.201+0000 7f0b47fff700 1 -- 172.21.15.27:0/83892428 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0b480d1030 msgr2=0x7f0b48157ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:29.209 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.201+0000 7f0b47fff700 1 --2- 172.21.15.27:0/83892428 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0b480d1030 0x7f0b48157ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:29.209 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.201+0000 7f0b47fff700 1 -- 172.21.15.27:0/83892428 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0b480cdc00 msgr2=0x7f0b48169a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:29.209 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.201+0000 7f0b47fff700 1 --2- 172.21.15.27:0/83892428 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0b480cdc00 0x7f0b48169a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:29.209 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.201+0000 7f0b47fff700 1 -- 172.21.15.27:0/83892428 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b30009420 con 0x7f0b480c6bf0 2023-10-06T01:41:29.209 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.201+0000 7f0b47fff700 1 --2- 172.21.15.27:0/83892428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b480c6bf0 0x7f0b48068f30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0b38012000 tx=0x7f0b38016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:29.209 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.202+0000 7f0b3ffff700 1 -- 172.21.15.27:0/83892428 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b3801a630 con 0x7f0b480c6bf0 2023-10-06T01:41:29.209 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.202+0000 7f0b3ffff700 1 -- 172.21.15.27:0/83892428 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b3801a7b0 con 0x7f0b480c6bf0 2023-10-06T01:41:29.210 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.202+0000 7f0b4e2bc700 1 -- 172.21.15.27:0/83892428 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b380093c0 con 0x7f0b480c6bf0 2023-10-06T01:41:29.210 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.202+0000 7f0b3ffff700 1 -- 172.21.15.27:0/83892428 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b3801ab70 con 0x7f0b480c6bf0 2023-10-06T01:41:29.210 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.202+0000 7f0b4e2bc700 1 -- 172.21.15.27:0/83892428 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b48068730 con 0x7f0b480c6bf0 2023-10-06T01:41:29.210 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.202+0000 7f0b4e2bc700 1 -- 172.21.15.27:0/83892428 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b48068d80 con 0x7f0b480c6bf0 2023-10-06T01:41:29.210 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.203+0000 7f0b3ffff700 1 -- 172.21.15.27:0/83892428 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0b38011420 con 0x7f0b480c6bf0 2023-10-06T01:41:29.210 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.203+0000 7f0b3ffff700 1 --2- 172.21.15.27:0/83892428 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0b28066f30 0x7f0b280693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:29.210 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.203+0000 7f0b3ffff700 1 -- 172.21.15.27:0/83892428 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f0b38085aa0 con 0x7f0b480c6bf0 2023-10-06T01:41:29.210 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.203+0000 7f0b477fe700 1 --2- 172.21.15.27:0/83892428 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0b28066f30 0x7f0b280693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:29.211 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.204+0000 7f0b477fe700 1 --2- 172.21.15.27:0/83892428 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0b28066f30 0x7f0b280693e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0b480d0ac0 tx=0x7f0b300091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:29.211 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.208+0000 7f0b3ffff700 1 -- 172.21.15.27:0/83892428 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0b38052bb0 con 0x7f0b480c6bf0 2023-10-06T01:41:29.405 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.404+0000 7f0b4e2bc700 1 -- 172.21.15.27:0/83892428 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b48064050 con 0x7f0b480c6bf0 2023-10-06T01:41:29.406 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.405+0000 7f0b3ffff700 1 -- 172.21.15.27:0/83892428 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16866 (secure 0 0 0) 0x7f0b3802b020 con 0x7f0b480c6bf0 2023-10-06T01:41:29.406 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:29.406 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":20,"fsid":"20a7bff0-74f8-4f2f-b6c6-899e804bc965","created":"2023-10-06T01:40:54.474215+0000","modified":"2023-10-06T01:41:14.324546+0000","last_up_change":"2023-10-06T01:41:05.242847+0000","last_in_change":"2023-10-06T01:40:56.890960+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-10-06T01:41:05.685386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-10-06T01:41:06.092995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T01:41:10.434257+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T01:41:11.700262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b902c86d-800c-40c5-b152-b545e50d02c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":90644},{"type":"v1","addr":"172.21.15.27:6801","nonce":90644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":90644},{"type":"v1","addr":"172.21.15.27:6803","nonce":90644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":90644},{"type":"v1","addr":"172.21.15.27:6807","nonce":90644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":90644},{"type":"v1","addr":"172.21.15.27:6805","nonce":90644}]},"public_addr":"172.21.15.27:6801/90644","cluster_addr":"172.21.15.27:6803/90644","heartbeat_back_addr":"172.21.15.27:6807/90644","heartbeat_front_addr":"172.21.15.27:6805/90644","state":["exists","up"]},{"osd":1,"uuid":"5b4a70e5-b06e-48cd-bfc4-7962c5d05386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":90643},{"type":"v1","addr":"172.21.15.27:6809","nonce":90643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":90643},{"type":"v1","addr":"172.21.15.27:6811","nonce":90643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":90643},{"type":"v1","addr":"172.21.15.27:6815","nonce":90643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":90643},{"type":"v1","addr":"172.21.15.27:6813","nonce":90643}]},"public_addr":"172.21.15.27:6809/90643","cluster_addr":"172.21.15.27:6811/90643","heartbeat_back_addr":"172.21.15.27:6815/90643","heartbeat_front_addr":"172.21.15.27:6813/90643","state":["exists","up"]},{"osd":2,"uuid":"22d9de46-4073-4336-9122-5227e601eca0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":90645},{"type":"v1","addr":"172.21.15.27:6817","nonce":90645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":90645},{"type":"v1","addr":"172.21.15.27:6819","nonce":90645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":90645},{"type":"v1","addr":"172.21.15.27:6823","nonce":90645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":90645},{"type":"v1","addr":"172.21.15.27:6821","nonce":90645}]},"public_addr":"172.21.15.27:6817/90645","cluster_addr":"172.21.15.27:6819/90645","heartbeat_back_addr":"172.21.15.27:6823/90645","heartbeat_front_addr":"172.21.15.27:6821/90645","state":["exists","up"]},{"osd":3,"uuid":"438bb2d3-34e2-4b7c-9734-8360c1b67650","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":90668},{"type":"v1","addr":"172.21.15.27:6825","nonce":90668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":90668},{"type":"v1","addr":"172.21.15.27:6827","nonce":90668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":90668},{"type":"v1","addr":"172.21.15.27:6831","nonce":90668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":90668},{"type":"v1","addr":"172.21.15.27:6829","nonce":90668}]},"public_addr":"172.21.15.27:6825/90668","cluster_addr":"172.21.15.27:6827/90668","heartbeat_back_addr":"172.21.15.27:6831/90668","heartbeat_front_addr":"172.21.15.27:6829/90668","state":["exists","up"]},{"osd":4,"uuid":"e1956a8e-e0f8-4e7c-acbb-40037415a354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":90863},{"type":"v1","addr":"172.21.15.124:6801","nonce":90863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":90863},{"type":"v1","addr":"172.21.15.124:6803","nonce":90863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":90863},{"type":"v1","addr":"172.21.15.124:6807","nonce":90863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":90863},{"type":"v1","addr":"172.21.15.124:6805","nonce":90863}]},"public_addr":"172.21.15.124:6801/90863","cluster_addr":"172.21.15.124:6803/90863","heartbeat_back_addr":"172.21.15.124:6807/90863","heartbeat_front_addr":"172.21.15.124:6805/90863","state":["exists","up"]},{"osd":5,"uuid":"306c59d4-846d-4b73-b1c9-051bec9e9368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":90885},{"type":"v1","addr":"172.21.15.124:6817","nonce":90885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":90885},{"type":"v1","addr":"172.21.15.124:6819","nonce":90885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":90885},{"type":"v1","addr":"172.21.15.124:6823","nonce":90885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":90885},{"type":"v1","addr":"172.21.15.124:6821","nonce":90885}]},"public_addr":"172.21.15.124:6817/90885","cluster_addr":"172.21.15.124:6819/90885","heartbeat_back_addr":"172.21.15.124:6823/90885","heartbeat_front_addr":"172.21.15.124:6821/90885","state":["exists","up"]},{"osd":6,"uuid":"15463e53-ac8d-47d1-b4f6-c0d31f29e4a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":90864},{"type":"v1","addr":"172.21.15.124:6809","nonce":90864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":90864},{"type":"v1","addr":"172.21.15.124:6811","nonce":90864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":90864},{"type":"v1","addr":"172.21.15.124:6815","nonce":90864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":90864},{"type":"v1","addr":"172.21.15.124:6813","nonce":90864}]},"public_addr":"172.21.15.124:6809/90864","cluster_addr":"172.21.15.124:6811/90864","heartbeat_back_addr":"172.21.15.124:6815/90864","heartbeat_front_addr":"172.21.15.124:6813/90864","state":["exists","up"]},{"osd":7,"uuid":"ae463a90-a9d3-498f-86fa-47a35b3fe889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":90894},{"type":"v1","addr":"172.21.15.124:6825","nonce":90894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":90894},{"type":"v1","addr":"172.21.15.124:6827","nonce":90894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":90894},{"type":"v1","addr":"172.21.15.124:6831","nonce":90894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":90894},{"type":"v1","addr":"172.21.15.124:6829","nonce":90894}]},"public_addr":"172.21.15.124:6825/90894","cluster_addr":"172.21.15.124:6827/90894","heartbeat_back_addr":"172.21.15.124:6831/90894","heartbeat_front_addr":"172.21.15.124:6829/90894","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.827704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.839951+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.032230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.050605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.987908+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.031290+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.080707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.053692+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T01:41:29.408 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.407+0000 7f0b4e2bc700 1 -- 172.21.15.27:0/83892428 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0b28066f30 msgr2=0x7f0b280693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:29.409 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.407+0000 7f0b4e2bc700 1 --2- 172.21.15.27:0/83892428 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0b28066f30 0x7f0b280693e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0b480d0ac0 tx=0x7f0b300091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:29.409 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.407+0000 7f0b4e2bc700 1 -- 172.21.15.27:0/83892428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b480c6bf0 msgr2=0x7f0b48068f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:29.409 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.407+0000 7f0b4e2bc700 1 --2- 172.21.15.27:0/83892428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b480c6bf0 0x7f0b48068f30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0b38012000 tx=0x7f0b38016000 comp rx=0 tx=0).stop 2023-10-06T01:41:29.409 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.407+0000 7f0b4e2bc700 1 -- 172.21.15.27:0/83892428 shutdown_connections 2023-10-06T01:41:29.409 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.407+0000 7f0b4e2bc700 1 --2- 172.21.15.27:0/83892428 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0b28066f30 0x7f0b280693e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:29.409 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.407+0000 7f0b4e2bc700 1 --2- 172.21.15.27:0/83892428 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0b480d1030 0x7f0b48157ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:29.409 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.407+0000 7f0b4e2bc700 1 --2- 172.21.15.27:0/83892428 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0b480c6bf0 0x7f0b48068f30 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:29.409 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.407+0000 7f0b4e2bc700 1 --2- 172.21.15.27:0/83892428 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0b480cdc00 0x7f0b48169a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:29.409 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.407+0000 7f0b4e2bc700 1 -- 172.21.15.27:0/83892428 >> 172.21.15.27:0/83892428 conn(0x7f0b48099990 msgr2=0x7f0b48159140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:29.410 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.407+0000 7f0b4e2bc700 1 -- 172.21.15.27:0/83892428 shutdown_connections 2023-10-06T01:41:29.410 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.407+0000 7f0b4e2bc700 1 -- 172.21.15.27:0/83892428 wait complete. 2023-10-06T01:41:29.420 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-10-06T01:41:05.685386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-10-06T01:41:06.092995+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-06T01:41:10.434257+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-10-06T01:41:11.700262+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-06T01:41:29.421 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-06T01:41:29.543 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.542+0000 7f07ccfdd700 1 Processor -- start 2023-10-06T01:41:29.543 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.542+0000 7f07ccfdd700 1 -- start start 2023-10-06T01:41:29.546 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.543+0000 7f07ccfdd700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f07c8142e90 0x7f07c8069810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:29.546 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.543+0000 7f07ccfdd700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f07c8069d50 0x7f07c815d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:29.546 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.543+0000 7f07ccfdd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f07c815de80 0x7f07c815d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:29.546 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.543+0000 7f07ccfdd700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07c8069380 con 0x7f07c815de80 2023-10-06T01:41:29.546 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.543+0000 7f07ccfdd700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07c8068770 con 0x7f07c8069d50 2023-10-06T01:41:29.546 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.543+0000 7f07ccfdd700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07c8068a50 con 0x7f07c8142e90 2023-10-06T01:41:29.546 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.543+0000 7f07c7fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f07c815de80 0x7f07c815d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:29.546 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.543+0000 7f07c7fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f07c815de80 0x7f07c815d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52380/0 (socket says 172.21.15.27:52380) 2023-10-06T01:41:29.546 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.543+0000 7f07c7fff700 1 -- 172.21.15.27:0/4250289866 learned_addr learned my addr 172.21.15.27:0/4250289866 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:29.546 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.543+0000 7f07c77fe700 1 --2- 172.21.15.27:0/4250289866 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f07c8142e90 0x7f07c8069810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:29.547 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.543+0000 7f07c6ffd700 1 --2- 172.21.15.27:0/4250289866 >> v2:172.21.15.124:3300/0 conn(0x7f07c8069d50 0x7f07c815d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:29.547 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.543+0000 7f07c7fff700 1 -- 172.21.15.27:0/4250289866 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f07c8142e90 msgr2=0x7f07c8069810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:29.547 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.543+0000 7f07c7fff700 1 --2- 172.21.15.27:0/4250289866 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f07c8142e90 0x7f07c8069810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:29.547 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.543+0000 7f07c7fff700 1 -- 172.21.15.27:0/4250289866 >> v2:172.21.15.124:3300/0 conn(0x7f07c8069d50 msgr2=0x7f07c815d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:29.547 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.543+0000 7f07c7fff700 1 --2- 172.21.15.27:0/4250289866 >> v2:172.21.15.124:3300/0 conn(0x7f07c8069d50 0x7f07c815d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:29.547 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.543+0000 7f07c7fff700 1 -- 172.21.15.27:0/4250289866 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c8068d30 con 0x7f07c815de80 2023-10-06T01:41:29.547 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.543+0000 7f07c7fff700 1 --2- 172.21.15.27:0/4250289866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07c815de80 0x7f07c815d7d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f07c8071360 tx=0x7f07c0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48ac36daa1dafbcd server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:29.547 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.544+0000 7f07c67fc700 1 -- 172.21.15.27:0/4250289866 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c000e070 con 0x7f07c815de80 2023-10-06T01:41:29.547 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.544+0000 7f07c67fc700 1 -- 172.21.15.27:0/4250289866 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07c000b970 con 0x7f07c815de80 2023-10-06T01:41:29.547 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.544+0000 7f07c67fc700 1 -- 172.21.15.27:0/4250289866 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c00092e0 con 0x7f07c815de80 2023-10-06T01:41:29.547 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.544+0000 7f07ccfdd700 1 -- 172.21.15.27:0/4250289866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07c815de80 msgr2=0x7f07c815d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:29.548 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.544+0000 7f07ccfdd700 1 --2- 172.21.15.27:0/4250289866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07c815de80 0x7f07c815d7d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f07c8071360 tx=0x7f07c0009840 comp rx=0 tx=0).stop 2023-10-06T01:41:29.548 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.544+0000 7f07ccfdd700 1 -- 172.21.15.27:0/4250289866 shutdown_connections 2023-10-06T01:41:29.548 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.544+0000 7f07ccfdd700 1 --2- 172.21.15.27:0/4250289866 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07c815de80 0x7f07c815d7d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:29.548 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.544+0000 7f07ccfdd700 1 --2- 172.21.15.27:0/4250289866 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f07c8142e90 0x7f07c8069810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:29.548 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.544+0000 7f07ccfdd700 1 --2- 172.21.15.27:0/4250289866 >> v2:172.21.15.124:3300/0 conn(0x7f07c8069d50 0x7f07c815d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:29.548 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.544+0000 7f07ccfdd700 1 -- 172.21.15.27:0/4250289866 >> 172.21.15.27:0/4250289866 conn(0x7f07c80ccfe0 msgr2=0x7f07c80cd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:29.548 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.544+0000 7f07ccfdd700 1 -- 172.21.15.27:0/4250289866 shutdown_connections 2023-10-06T01:41:29.548 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.544+0000 7f07ccfdd700 1 -- 172.21.15.27:0/4250289866 wait complete. 2023-10-06T01:41:29.551 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.546+0000 7f07ccfdd700 1 Processor -- start 2023-10-06T01:41:29.551 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.546+0000 7f07ccfdd700 1 -- start start 2023-10-06T01:41:29.551 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.546+0000 7f07ccfdd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07c8069d50 0x7f07c816d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:29.551 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.546+0000 7f07ccfdd700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f07c8142e90 0x7f07c815a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:29.551 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.546+0000 7f07ccfdd700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07c815de80 0x7f07c8158d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:29.551 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.546+0000 7f07ccfdd700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c816e320 con 0x7f07c8069d50 2023-10-06T01:41:29.551 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.546+0000 7f07ccfdd700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c8169ff0 con 0x7f07c815de80 2023-10-06T01:41:29.551 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.546+0000 7f07ccfdd700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07c816da80 con 0x7f07c8142e90 2023-10-06T01:41:29.551 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.546+0000 7f07c77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07c8069d50 0x7f07c816d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:29.551 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.546+0000 7f07c77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07c8069d50 0x7f07c816d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52392/0 (socket says 172.21.15.27:52392) 2023-10-06T01:41:29.552 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.546+0000 7f07c77fe700 1 -- 172.21.15.27:0/816877530 learned_addr learned my addr 172.21.15.27:0/816877530 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:29.552 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.547+0000 7f07c6ffd700 1 --2- 172.21.15.27:0/816877530 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f07c8142e90 0x7f07c815a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:29.552 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.547+0000 7f07c7fff700 1 --2- 172.21.15.27:0/816877530 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07c815de80 0x7f07c8158d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:29.552 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.547+0000 7f07c77fe700 1 -- 172.21.15.27:0/816877530 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f07c8142e90 msgr2=0x7f07c815a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:29.552 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.547+0000 7f07c77fe700 1 --2- 172.21.15.27:0/816877530 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f07c8142e90 0x7f07c815a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:29.552 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.547+0000 7f07c77fe700 1 -- 172.21.15.27:0/816877530 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07c815de80 msgr2=0x7f07c8158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:29.552 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.547+0000 7f07c77fe700 1 --2- 172.21.15.27:0/816877530 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07c815de80 0x7f07c8158d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:29.552 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.547+0000 7f07c77fe700 1 -- 172.21.15.27:0/816877530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c0009ae0 con 0x7f07c8069d50 2023-10-06T01:41:29.552 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.547+0000 7f07c77fe700 1 --2- 172.21.15.27:0/816877530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07c8069d50 0x7f07c816d3c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f07b800ec80 tx=0x7f07b80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:29.552 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.547+0000 7f07af7fe700 1 -- 172.21.15.27:0/816877530 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b800c850 con 0x7f07c8069d50 2023-10-06T01:41:29.552 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.547+0000 7f07af7fe700 1 -- 172.21.15.27:0/816877530 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07b800c9b0 con 0x7f07c8069d50 2023-10-06T01:41:29.553 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.547+0000 7f07ccfdd700 1 -- 172.21.15.27:0/816877530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07c816dd60 con 0x7f07c8069d50 2023-10-06T01:41:29.553 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.547+0000 7f07af7fe700 1 -- 172.21.15.27:0/816877530 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b800cd50 con 0x7f07c8069d50 2023-10-06T01:41:29.553 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.547+0000 7f07ccfdd700 1 -- 172.21.15.27:0/816877530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07c8159240 con 0x7f07c8069d50 2023-10-06T01:41:29.553 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.548+0000 7f07ccfdd700 1 -- 172.21.15.27:0/816877530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07c8159a60 con 0x7f07c8069d50 2023-10-06T01:41:29.553 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.549+0000 7f07af7fe700 1 -- 172.21.15.27:0/816877530 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f07b8013680 con 0x7f07c8069d50 2023-10-06T01:41:29.553 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.549+0000 7f07af7fe700 1 --2- 172.21.15.27:0/816877530 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f07b4066f30 0x7f07b40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:29.553 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.549+0000 7f07af7fe700 1 -- 172.21.15.27:0/816877530 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f07b80848c0 con 0x7f07c8069d50 2023-10-06T01:41:29.553 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.549+0000 7f07c6ffd700 1 --2- 172.21.15.27:0/816877530 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f07b4066f30 0x7f07b40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:29.553 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.550+0000 7f07c6ffd700 1 --2- 172.21.15.27:0/816877530 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f07b4066f30 0x7f07b40693e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f07c815b450 tx=0x7f07bc0091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:29.554 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.553+0000 7f07af7fe700 1 -- 172.21.15.27:0/816877530 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f07b80519d0 con 0x7f07c8069d50 2023-10-06T01:41:29.747 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.745+0000 7f07ccfdd700 1 -- 172.21.15.27:0/816877530 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f07c816a2d0 con 0x7f07c8069d50 2023-10-06T01:41:29.747 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.746+0000 7f07af7fe700 1 -- 172.21.15.27:0/816877530 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f07b8054ff0 con 0x7f07c8069d50 2023-10-06T01:41:29.747 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 8 2023-10-06T01:41:29.749 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.748+0000 7f07ccfdd700 1 -- 172.21.15.27:0/816877530 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f07b4066f30 msgr2=0x7f07b40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:29.749 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.748+0000 7f07ccfdd700 1 --2- 172.21.15.27:0/816877530 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f07b4066f30 0x7f07b40693e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f07c815b450 tx=0x7f07bc0091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:29.750 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.748+0000 7f07ccfdd700 1 -- 172.21.15.27:0/816877530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07c8069d50 msgr2=0x7f07c816d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:29.750 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.748+0000 7f07ccfdd700 1 --2- 172.21.15.27:0/816877530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07c8069d50 0x7f07c816d3c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f07b800ec80 tx=0x7f07b80091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:29.750 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.748+0000 7f07ccfdd700 1 -- 172.21.15.27:0/816877530 shutdown_connections 2023-10-06T01:41:29.750 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.748+0000 7f07ccfdd700 1 --2- 172.21.15.27:0/816877530 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f07b4066f30 0x7f07b40693e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:29.750 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.748+0000 7f07ccfdd700 1 --2- 172.21.15.27:0/816877530 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f07c815de80 0x7f07c8158d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:29.750 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.748+0000 7f07ccfdd700 1 --2- 172.21.15.27:0/816877530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f07c8069d50 0x7f07c816d3c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:29.750 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.748+0000 7f07ccfdd700 1 --2- 172.21.15.27:0/816877530 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f07c8142e90 0x7f07c815a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:29.751 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.748+0000 7f07ccfdd700 1 -- 172.21.15.27:0/816877530 >> 172.21.15.27:0/816877530 conn(0x7f07c80ccfe0 msgr2=0x7f07c813fdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:29.751 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.748+0000 7f07ccfdd700 1 -- 172.21.15.27:0/816877530 shutdown_connections 2023-10-06T01:41:29.751 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.748+0000 7f07ccfdd700 1 -- 172.21.15.27:0/816877530 wait complete. 2023-10-06T01:41:29.761 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-06T01:41:29.882 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.881+0000 7f323d2cb700 1 Processor -- start 2023-10-06T01:41:29.883 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.881+0000 7f323d2cb700 1 -- start start 2023-10-06T01:41:29.885 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.882+0000 7f323d2cb700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f3238142e90 0x7f3238069810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:29.885 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.882+0000 7f323d2cb700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3238069380 con 0x7f323815d9a0 2023-10-06T01:41:29.885 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.882+0000 7f323d2cb700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32380b63a0 con 0x7f3238069d50 2023-10-06T01:41:29.885 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.882+0000 7f323d2cb700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3238068770 con 0x7f3238142e90 2023-10-06T01:41:29.885 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.882+0000 7f323caca700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f323815d9a0 0x7f32380b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57056/0 (socket says 172.21.15.27:57056) 2023-10-06T01:41:29.885 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.882+0000 7f323caca700 1 -- 172.21.15.27:0/754777085 learned_addr learned my addr 172.21.15.27:0/754777085 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:29.885 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.882+0000 7f3237fff700 1 --2- 172.21.15.27:0/754777085 >> v2:172.21.15.124:3300/0 conn(0x7f3238142e90 0x7f3238069810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:29.885 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.882+0000 7f3236ffd700 1 -- 172.21.15.27:0/754777085 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1781517762 0 0) 0x7f3238069380 con 0x7f323815d9a0 2023-10-06T01:41:29.885 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.882+0000 7f3236ffd700 1 -- 172.21.15.27:0/754777085 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3220003570 con 0x7f323815d9a0 2023-10-06T01:41:29.886 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.883+0000 7f3236ffd700 1 -- 172.21.15.27:0/754777085 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3414412977 0 0) 0x7f32380b63a0 con 0x7f3238069d50 2023-10-06T01:41:29.886 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.883+0000 7f3236ffd700 1 -- 172.21.15.27:0/754777085 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3238069380 con 0x7f3238069d50 2023-10-06T01:41:29.886 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.883+0000 7f3236ffd700 1 -- 172.21.15.27:0/754777085 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 427166322 0 0) 0x7f3220003570 con 0x7f323815d9a0 2023-10-06T01:41:29.886 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.883+0000 7f3237fff700 1 -- 172.21.15.27:0/754777085 >> v1:172.21.15.124:6789/0 conn(0x7f3238069d50 legacy=0x7f323815d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:29.886 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.883+0000 7f3237fff700 1 -- 172.21.15.27:0/754777085 >> v1:172.21.15.27:6789/0 conn(0x7f323815d9a0 legacy=0x7f32380b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:29.886 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.883+0000 7f3237fff700 1 -- 172.21.15.27:0/754777085 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3238068a50 con 0x7f3238142e90 2023-10-06T01:41:29.886 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.883+0000 7f3237fff700 1 --2- 172.21.15.27:0/754777085 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3238142e90 0x7f3238069810 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f32280098e0 tx=0x7f32280091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bcbc18b04f04434e server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:29.886 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.883+0000 7f3236ffd700 1 -- 172.21.15.27:0/754777085 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32300017f0 con 0x7f3238142e90 2023-10-06T01:41:29.886 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.883+0000 7f3236ffd700 1 -- 172.21.15.27:0/754777085 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3228011420 con 0x7f3238142e90 2023-10-06T01:41:29.886 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.883+0000 7f3236ffd700 1 -- 172.21.15.27:0/754777085 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32280117c0 con 0x7f3238142e90 2023-10-06T01:41:29.886 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.883+0000 7f323d2cb700 1 -- 172.21.15.27:0/754777085 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3238142e90 msgr2=0x7f3238069810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:29.887 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.883+0000 7f323d2cb700 1 --2- 172.21.15.27:0/754777085 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3238142e90 0x7f3238069810 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f32280098e0 tx=0x7f32280091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:29.887 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.883+0000 7f323d2cb700 1 -- 172.21.15.27:0/754777085 shutdown_connections 2023-10-06T01:41:29.887 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.883+0000 7f323d2cb700 1 --2- 172.21.15.27:0/754777085 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3238142e90 0x7f3238069810 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:29.887 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.883+0000 7f323d2cb700 1 -- 172.21.15.27:0/754777085 >> 172.21.15.27:0/754777085 conn(0x7f32380ccfe0 msgr2=0x7f32380cd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:29.887 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.884+0000 7f323d2cb700 1 -- 172.21.15.27:0/754777085 shutdown_connections 2023-10-06T01:41:29.887 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.884+0000 7f323d2cb700 1 -- 172.21.15.27:0/754777085 wait complete. 2023-10-06T01:41:29.889 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.885+0000 7f323d2cb700 1 Processor -- start 2023-10-06T01:41:29.889 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.885+0000 7f323d2cb700 1 -- start start 2023-10-06T01:41:29.889 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.885+0000 7f323d2cb700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3238069d50 0x7f3238158c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:29.889 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.885+0000 7f323d2cb700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3238142e90 0x7f32380cc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:29.890 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.886+0000 7f323d2cb700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f323815d9a0 0x7f32380c5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:29.890 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.886+0000 7f323d2cb700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32380cbd30 con 0x7f323815d9a0 2023-10-06T01:41:29.890 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.886+0000 7f323d2cb700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32381591b0 con 0x7f3238069d50 2023-10-06T01:41:29.890 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.886+0000 7f323d2cb700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32380cb120 con 0x7f3238142e90 2023-10-06T01:41:29.890 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.886+0000 7f323caca700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f323815d9a0 0x7f32380c5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:29.890 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.886+0000 7f323caca700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f323815d9a0 0x7f32380c5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52404/0 (socket says 172.21.15.27:52404) 2023-10-06T01:41:29.890 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.886+0000 7f323caca700 1 -- 172.21.15.27:0/1661344557 learned_addr learned my addr 172.21.15.27:0/1661344557 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:29.890 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.886+0000 7f3237fff700 1 --2- 172.21.15.27:0/1661344557 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3238069d50 0x7f3238158c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:29.890 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.886+0000 7f32377fe700 1 --2- 172.21.15.27:0/1661344557 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3238142e90 0x7f32380cc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:29.890 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.886+0000 7f323caca700 1 -- 172.21.15.27:0/1661344557 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3238142e90 msgr2=0x7f32380cc1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:29.890 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.886+0000 7f323caca700 1 --2- 172.21.15.27:0/1661344557 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3238142e90 0x7f32380cc1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:29.891 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.886+0000 7f323caca700 1 -- 172.21.15.27:0/1661344557 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3238069d50 msgr2=0x7f3238158c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:29.891 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.886+0000 7f323caca700 1 --2- 172.21.15.27:0/1661344557 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3238069d50 0x7f3238158c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:29.891 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.886+0000 7f323caca700 1 -- 172.21.15.27:0/1661344557 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3228009420 con 0x7f323815d9a0 2023-10-06T01:41:29.891 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.886+0000 7f323caca700 1 --2- 172.21.15.27:0/1661344557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f323815d9a0 0x7f32380c5310 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3230001560 tx=0x7f3230007d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:29.891 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.886+0000 7f321ffff700 1 -- 172.21.15.27:0/1661344557 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3230004dc0 con 0x7f323815d9a0 2023-10-06T01:41:29.891 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.886+0000 7f321ffff700 1 -- 172.21.15.27:0/1661344557 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f323000a590 con 0x7f323815d9a0 2023-10-06T01:41:29.891 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.886+0000 7f323d2cb700 1 -- 172.21.15.27:0/1661344557 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32380cb400 con 0x7f323815d9a0 2023-10-06T01:41:29.891 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.886+0000 7f321ffff700 1 -- 172.21.15.27:0/1661344557 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f323000a930 con 0x7f323815d9a0 2023-10-06T01:41:29.891 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.886+0000 7f323d2cb700 1 -- 172.21.15.27:0/1661344557 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32380cb9c0 con 0x7f323815d9a0 2023-10-06T01:41:29.891 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.887+0000 7f323d2cb700 1 -- 172.21.15.27:0/1661344557 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32380cbee0 con 0x7f323815d9a0 2023-10-06T01:41:29.891 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.888+0000 7f321ffff700 1 -- 172.21.15.27:0/1661344557 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f323000ace0 con 0x7f323815d9a0 2023-10-06T01:41:29.891 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.888+0000 7f321ffff700 1 --2- 172.21.15.27:0/1661344557 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3224066f30 0x7f32240693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:29.892 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.888+0000 7f321ffff700 1 -- 172.21.15.27:0/1661344557 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f32300840a0 con 0x7f323815d9a0 2023-10-06T01:41:29.892 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.888+0000 7f3237fff700 1 --2- 172.21.15.27:0/1661344557 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3224066f30 0x7f32240693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:29.892 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.889+0000 7f3237fff700 1 --2- 172.21.15.27:0/1661344557 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3224066f30 0x7f32240693e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f32280093f0 tx=0x7f3228016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:29.893 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:29.892+0000 7f321ffff700 1 -- 172.21.15.27:0/1661344557 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f32380cbee0 con 0x7f323815d9a0 2023-10-06T01:41:30.083 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.082+0000 7f323d2cb700 1 -- 172.21.15.27:0/1661344557 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f323804c3c0 con 0x7f323815d9a0 2023-10-06T01:41:30.084 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.083+0000 7f321ffff700 1 -- 172.21.15.27:0/1661344557 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f32300180a0 con 0x7f323815d9a0 2023-10-06T01:41:30.084 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2023-10-06T01:41:30.086 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.084+0000 7f323d2cb700 1 -- 172.21.15.27:0/1661344557 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3224066f30 msgr2=0x7f32240693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:30.086 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.084+0000 7f323d2cb700 1 --2- 172.21.15.27:0/1661344557 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3224066f30 0x7f32240693e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f32280093f0 tx=0x7f3228016000 comp rx=0 tx=0).stop 2023-10-06T01:41:30.086 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.084+0000 7f323d2cb700 1 -- 172.21.15.27:0/1661344557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f323815d9a0 msgr2=0x7f32380c5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:30.086 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.084+0000 7f323d2cb700 1 --2- 172.21.15.27:0/1661344557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f323815d9a0 0x7f32380c5310 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f3230001560 tx=0x7f3230007d20 comp rx=0 tx=0).stop 2023-10-06T01:41:30.086 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.085+0000 7f323d2cb700 1 -- 172.21.15.27:0/1661344557 shutdown_connections 2023-10-06T01:41:30.087 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.085+0000 7f323d2cb700 1 --2- 172.21.15.27:0/1661344557 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3224066f30 0x7f32240693e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:30.087 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.085+0000 7f323d2cb700 1 --2- 172.21.15.27:0/1661344557 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f323815d9a0 0x7f32380c5310 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:30.087 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.085+0000 7f323d2cb700 1 --2- 172.21.15.27:0/1661344557 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3238069d50 0x7f3238158c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:30.087 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.085+0000 7f323d2cb700 1 --2- 172.21.15.27:0/1661344557 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3238142e90 0x7f32380cc1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:30.087 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.085+0000 7f323d2cb700 1 -- 172.21.15.27:0/1661344557 >> 172.21.15.27:0/1661344557 conn(0x7f32380ccfe0 msgr2=0x7f323813ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:30.087 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.085+0000 7f323d2cb700 1 -- 172.21.15.27:0/1661344557 shutdown_connections 2023-10-06T01:41:30.087 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.085+0000 7f323d2cb700 1 -- 172.21.15.27:0/1661344557 wait complete. 2023-10-06T01:41:30.098 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-06T01:41:30.220 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.219+0000 7fec496fa700 1 Processor -- start 2023-10-06T01:41:30.220 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.219+0000 7fec496fa700 1 -- start start 2023-10-06T01:41:30.222 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.220+0000 7fec496fa700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fec441570c0 0x7fec440b3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:30.222 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.220+0000 7fec496fa700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec440b5b20 con 0x7fec4415da60 2023-10-06T01:41:30.222 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.220+0000 7fec496fa700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec440b5dc0 con 0x7fec44157490 2023-10-06T01:41:30.222 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.220+0000 7fec496fa700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec440b5920 con 0x7fec441570c0 2023-10-06T01:41:30.223 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.220+0000 7fec42ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fec441570c0 0x7fec440b3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:30.223 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.220+0000 7fec437fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fec4415da60 0x7fec441426c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57072/0 (socket says 172.21.15.27:57072) 2023-10-06T01:41:30.223 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.220+0000 7fec437fe700 1 -- 172.21.15.27:0/2106826414 learned_addr learned my addr 172.21.15.27:0/2106826414 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:30.223 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.220+0000 7fec42ffd700 1 -- 172.21.15.27:0/2106826414 >> v1:172.21.15.124:6789/0 conn(0x7fec44157490 legacy=0x7fec4415d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:30.223 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.220+0000 7fec42ffd700 1 -- 172.21.15.27:0/2106826414 >> v1:172.21.15.27:6789/0 conn(0x7fec4415da60 legacy=0x7fec441426c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:30.223 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.220+0000 7fec41ffb700 1 -- 172.21.15.27:0/2106826414 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2019244563 0 0) 0x7fec440b5b20 con 0x7fec4415da60 2023-10-06T01:41:30.223 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.220+0000 7fec42ffd700 1 -- 172.21.15.27:0/2106826414 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec440b60a0 con 0x7fec441570c0 2023-10-06T01:41:30.223 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.220+0000 7fec42ffd700 1 --2- 172.21.15.27:0/2106826414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec441570c0 0x7fec440b3310 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fec340098e0 tx=0x7fec340091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe9530066eb70ca7 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:30.223 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.220+0000 7fec41ffb700 1 -- 172.21.15.27:0/2106826414 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec34009e80 con 0x7fec441570c0 2023-10-06T01:41:30.223 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.220+0000 7fec41ffb700 1 -- 172.21.15.27:0/2106826414 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec3400b860 con 0x7fec441570c0 2023-10-06T01:41:30.224 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.220+0000 7fec41ffb700 1 -- 172.21.15.27:0/2106826414 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec3400bbe0 con 0x7fec441570c0 2023-10-06T01:41:30.224 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.221+0000 7fec496fa700 1 -- 172.21.15.27:0/2106826414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec441570c0 msgr2=0x7fec440b3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:30.224 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.221+0000 7fec496fa700 1 --2- 172.21.15.27:0/2106826414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec441570c0 0x7fec440b3310 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fec340098e0 tx=0x7fec340091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:30.224 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.221+0000 7fec496fa700 1 -- 172.21.15.27:0/2106826414 shutdown_connections 2023-10-06T01:41:30.224 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.221+0000 7fec496fa700 1 --2- 172.21.15.27:0/2106826414 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec441570c0 0x7fec440b3310 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:30.224 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.221+0000 7fec496fa700 1 -- 172.21.15.27:0/2106826414 >> 172.21.15.27:0/2106826414 conn(0x7fec440ccfc0 msgr2=0x7fec440cd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:30.224 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.221+0000 7fec496fa700 1 -- 172.21.15.27:0/2106826414 shutdown_connections 2023-10-06T01:41:30.224 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.221+0000 7fec496fa700 1 -- 172.21.15.27:0/2106826414 wait complete. 2023-10-06T01:41:30.227 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.222+0000 7fec496fa700 1 Processor -- start 2023-10-06T01:41:30.227 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.222+0000 7fec496fa700 1 -- start start 2023-10-06T01:41:30.227 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.223+0000 7fec496fa700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec441570c0 0x7fec44148f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:30.227 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.223+0000 7fec496fa700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fec44157490 0x7fec441597f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:30.227 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.223+0000 7fec496fa700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec4415da60 0x7fec44159d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:30.227 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.223+0000 7fec496fa700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec440c5da0 con 0x7fec441570c0 2023-10-06T01:41:30.228 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.223+0000 7fec496fa700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec440c5470 con 0x7fec4415da60 2023-10-06T01:41:30.228 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.223+0000 7fec496fa700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec440c5750 con 0x7fec44157490 2023-10-06T01:41:30.228 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.223+0000 7fec42ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec441570c0 0x7fec44148f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:30.228 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.223+0000 7fec42ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec441570c0 0x7fec44148f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52422/0 (socket says 172.21.15.27:52422) 2023-10-06T01:41:30.228 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.223+0000 7fec42ffd700 1 -- 172.21.15.27:0/3824672574 learned_addr learned my addr 172.21.15.27:0/3824672574 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:30.228 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.223+0000 7fec427fc700 1 --2- 172.21.15.27:0/3824672574 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fec44157490 0x7fec441597f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:30.228 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.223+0000 7fec437fe700 1 --2- 172.21.15.27:0/3824672574 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec4415da60 0x7fec44159d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:30.228 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.223+0000 7fec42ffd700 1 -- 172.21.15.27:0/3824672574 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fec44157490 msgr2=0x7fec441597f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:30.228 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.223+0000 7fec42ffd700 1 --2- 172.21.15.27:0/3824672574 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fec44157490 0x7fec441597f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:30.229 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.223+0000 7fec42ffd700 1 -- 172.21.15.27:0/3824672574 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec4415da60 msgr2=0x7fec44159d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:30.229 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.223+0000 7fec42ffd700 1 --2- 172.21.15.27:0/3824672574 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec4415da60 0x7fec44159d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:30.229 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.223+0000 7fec42ffd700 1 -- 172.21.15.27:0/3824672574 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec34009420 con 0x7fec441570c0 2023-10-06T01:41:30.229 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.224+0000 7fec42ffd700 1 --2- 172.21.15.27:0/3824672574 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec441570c0 0x7fec44148f70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fec440b2f90 tx=0x7fec34017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:30.229 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.224+0000 7fec2affd700 1 -- 172.21.15.27:0/3824672574 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec3401f070 con 0x7fec441570c0 2023-10-06T01:41:30.229 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.224+0000 7fec2affd700 1 -- 172.21.15.27:0/3824672574 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec3400b9d0 con 0x7fec441570c0 2023-10-06T01:41:30.229 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.224+0000 7fec496fa700 1 -- 172.21.15.27:0/3824672574 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec440c5a30 con 0x7fec441570c0 2023-10-06T01:41:30.229 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.224+0000 7fec2affd700 1 -- 172.21.15.27:0/3824672574 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec34025670 con 0x7fec441570c0 2023-10-06T01:41:30.229 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.224+0000 7fec496fa700 1 -- 172.21.15.27:0/3824672574 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec44169940 con 0x7fec441570c0 2023-10-06T01:41:30.229 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.225+0000 7fec496fa700 1 -- 172.21.15.27:0/3824672574 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec44169b60 con 0x7fec441570c0 2023-10-06T01:41:30.229 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.225+0000 7fec2affd700 1 -- 172.21.15.27:0/3824672574 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fec34023070 con 0x7fec441570c0 2023-10-06T01:41:30.230 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.225+0000 7fec2affd700 1 --2- 172.21.15.27:0/3824672574 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fec30066f30 0x7fec300693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:30.230 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.225+0000 7fec2affd700 1 -- 172.21.15.27:0/3824672574 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fec34085c90 con 0x7fec441570c0 2023-10-06T01:41:30.230 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.226+0000 7fec427fc700 1 --2- 172.21.15.27:0/3824672574 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fec30066f30 0x7fec300693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:30.230 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.226+0000 7fec427fc700 1 --2- 172.21.15.27:0/3824672574 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fec30066f30 0x7fec300693e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fec4415a4e0 tx=0x7fec38009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:30.231 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.230+0000 7fec2affd700 1 -- 172.21.15.27:0/3824672574 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fec34052da0 con 0x7fec441570c0 2023-10-06T01:41:30.422 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.421+0000 7fec496fa700 1 -- 172.21.15.27:0/3824672574 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fec4404c3c0 con 0x7fec441570c0 2023-10-06T01:41:30.423 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.422+0000 7fec2affd700 1 -- 172.21.15.27:0/3824672574 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fec34015020 con 0x7fec441570c0 2023-10-06T01:41:30.423 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2023-10-06T01:41:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.424+0000 7fec496fa700 1 -- 172.21.15.27:0/3824672574 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fec30066f30 msgr2=0x7fec300693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.424+0000 7fec496fa700 1 --2- 172.21.15.27:0/3824672574 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fec30066f30 0x7fec300693e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fec4415a4e0 tx=0x7fec38009040 comp rx=0 tx=0).stop 2023-10-06T01:41:30.425 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.424+0000 7fec496fa700 1 -- 172.21.15.27:0/3824672574 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec441570c0 msgr2=0x7fec44148f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:30.426 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.424+0000 7fec496fa700 1 --2- 172.21.15.27:0/3824672574 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec441570c0 0x7fec44148f70 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fec440b2f90 tx=0x7fec34017000 comp rx=0 tx=0).stop 2023-10-06T01:41:30.426 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.424+0000 7fec496fa700 1 -- 172.21.15.27:0/3824672574 shutdown_connections 2023-10-06T01:41:30.426 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.424+0000 7fec496fa700 1 --2- 172.21.15.27:0/3824672574 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fec30066f30 0x7fec300693e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:30.426 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.424+0000 7fec496fa700 1 --2- 172.21.15.27:0/3824672574 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fec4415da60 0x7fec44159d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:30.426 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.424+0000 7fec496fa700 1 --2- 172.21.15.27:0/3824672574 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fec441570c0 0x7fec44148f70 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:30.426 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.424+0000 7fec496fa700 1 --2- 172.21.15.27:0/3824672574 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fec44157490 0x7fec441597f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:30.426 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.424+0000 7fec496fa700 1 -- 172.21.15.27:0/3824672574 >> 172.21.15.27:0/3824672574 conn(0x7fec440ccfc0 msgr2=0x7fec440b32f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:30.426 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.424+0000 7fec496fa700 1 -- 172.21.15.27:0/3824672574 shutdown_connections 2023-10-06T01:41:30.426 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.424+0000 7fec496fa700 1 -- 172.21.15.27:0/3824672574 wait complete. 2023-10-06T01:41:30.437 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-06T01:41:30.559 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.557+0000 7fc5df765700 1 Processor -- start 2023-10-06T01:41:30.559 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.558+0000 7fc5df765700 1 -- start start 2023-10-06T01:41:30.561 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.558+0000 7fc5df765700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc5d815dab0 0x7fc5d80cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:30.561 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.558+0000 7fc5df765700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d80b1110 con 0x7fc5d81598d0 2023-10-06T01:41:30.561 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.558+0000 7fc5df765700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d80b13b0 con 0x7fc5d8159500 2023-10-06T01:41:30.561 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.558+0000 7fc5df765700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5d80b1690 con 0x7fc5d815dab0 2023-10-06T01:41:30.561 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.558+0000 7fc5ddd02700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc5d815dab0 0x7fc5d80cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:30.561 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.558+0000 7fc5dcd00700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc5d81598d0 0x7fc5d815d3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57086/0 (socket says 172.21.15.27:57086) 2023-10-06T01:41:30.562 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.559+0000 7fc5ddd02700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc5d815dab0 0x7fc5d80cf6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52434/0 (socket says 172.21.15.27:52434) 2023-10-06T01:41:30.562 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.559+0000 7fc5dcd00700 1 -- 172.21.15.27:0/2116686778 learned_addr learned my addr 172.21.15.27:0/2116686778 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:30.562 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.559+0000 7fc5ddd02700 1 -- 172.21.15.27:0/2116686778 >> v1:172.21.15.124:6789/0 conn(0x7fc5d8159500 legacy=0x7fc5d80b3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:30.562 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.559+0000 7fc5ddd02700 1 -- 172.21.15.27:0/2116686778 >> v1:172.21.15.27:6789/0 conn(0x7fc5d81598d0 legacy=0x7fc5d815d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:30.562 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.559+0000 7fc5d7fff700 1 -- 172.21.15.27:0/2116686778 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4118915675 0 0) 0x7fc5d80b1110 con 0x7fc5d81598d0 2023-10-06T01:41:30.562 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.559+0000 7fc5ddd02700 1 -- 172.21.15.27:0/2116686778 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d80b1970 con 0x7fc5d815dab0 2023-10-06T01:41:30.562 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.559+0000 7fc5ddd02700 1 --2- 172.21.15.27:0/2116686778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5d815dab0 0x7fc5d80cf6c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8009e60 tx=0x7fc5c8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f966aeaa23138ebf server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:30.562 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.559+0000 7fc5d7fff700 1 -- 172.21.15.27:0/2116686778 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c800e070 con 0x7fc5d815dab0 2023-10-06T01:41:30.562 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.559+0000 7fc5d7fff700 1 -- 172.21.15.27:0/2116686778 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c800bc30 con 0x7fc5d815dab0 2023-10-06T01:41:30.562 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.559+0000 7fc5d7fff700 1 -- 172.21.15.27:0/2116686778 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c8012430 con 0x7fc5d815dab0 2023-10-06T01:41:30.562 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.560+0000 7fc5df765700 1 -- 172.21.15.27:0/2116686778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5d815dab0 msgr2=0x7fc5d80cf6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:30.563 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.560+0000 7fc5df765700 1 --2- 172.21.15.27:0/2116686778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5d815dab0 0x7fc5d80cf6c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8009e60 tx=0x7fc5c8009780 comp rx=0 tx=0).stop 2023-10-06T01:41:30.563 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.560+0000 7fc5df765700 1 -- 172.21.15.27:0/2116686778 shutdown_connections 2023-10-06T01:41:30.563 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.560+0000 7fc5df765700 1 --2- 172.21.15.27:0/2116686778 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5d815dab0 0x7fc5d80cf6c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:30.563 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.560+0000 7fc5df765700 1 -- 172.21.15.27:0/2116686778 >> 172.21.15.27:0/2116686778 conn(0x7fc5d8067630 msgr2=0x7fc5d8069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:30.563 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.560+0000 7fc5df765700 1 -- 172.21.15.27:0/2116686778 shutdown_connections 2023-10-06T01:41:30.563 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.560+0000 7fc5df765700 1 -- 172.21.15.27:0/2116686778 wait complete. 2023-10-06T01:41:30.565 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.561+0000 7fc5df765700 1 Processor -- start 2023-10-06T01:41:30.565 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.561+0000 7fc5df765700 1 -- start start 2023-10-06T01:41:30.566 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.562+0000 7fc5df765700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc5d8159500 0x7fc5d80b43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:30.566 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.562+0000 7fc5df765700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc5d81598d0 0x7fc5d80b4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:30.566 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.562+0000 7fc5df765700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5d815dab0 0x7fc5d8069050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:30.566 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.562+0000 7fc5df765700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d8068c00 con 0x7fc5d815dab0 2023-10-06T01:41:30.566 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.562+0000 7fc5df765700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d8152f20 con 0x7fc5d81598d0 2023-10-06T01:41:30.566 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.562+0000 7fc5df765700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d8152960 con 0x7fc5d8159500 2023-10-06T01:41:30.566 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.562+0000 7fc5ddd02700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5d815dab0 0x7fc5d8069050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:30.566 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.562+0000 7fc5ddd02700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5d815dab0 0x7fc5d8069050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52438/0 (socket says 172.21.15.27:52438) 2023-10-06T01:41:30.566 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.562+0000 7fc5ddd02700 1 -- 172.21.15.27:0/2932273231 learned_addr learned my addr 172.21.15.27:0/2932273231 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:30.566 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.562+0000 7fc5dd501700 1 --2- 172.21.15.27:0/2932273231 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc5d8159500 0x7fc5d80b43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:30.566 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.562+0000 7fc5dcd00700 1 --2- 172.21.15.27:0/2932273231 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc5d81598d0 0x7fc5d80b4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:30.566 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.562+0000 7fc5ddd02700 1 -- 172.21.15.27:0/2932273231 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc5d8159500 msgr2=0x7fc5d80b43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:30.567 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.562+0000 7fc5ddd02700 1 --2- 172.21.15.27:0/2932273231 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc5d8159500 0x7fc5d80b43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:30.567 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.562+0000 7fc5ddd02700 1 -- 172.21.15.27:0/2932273231 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc5d81598d0 msgr2=0x7fc5d80b4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:30.567 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.562+0000 7fc5ddd02700 1 --2- 172.21.15.27:0/2932273231 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc5d81598d0 0x7fc5d80b4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:30.567 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.562+0000 7fc5ddd02700 1 -- 172.21.15.27:0/2932273231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c8009a20 con 0x7fc5d815dab0 2023-10-06T01:41:30.567 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.562+0000 7fc5dd501700 1 --2- 172.21.15.27:0/2932273231 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc5d8159500 0x7fc5d80b43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:30.567 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.562+0000 7fc5dcd00700 1 --2- 172.21.15.27:0/2932273231 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc5d81598d0 0x7fc5d80b4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:30.567 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.562+0000 7fc5ddd02700 1 --2- 172.21.15.27:0/2932273231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5d815dab0 0x7fc5d8069050 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8088d00 tx=0x7fc5c8019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:30.567 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.562+0000 7fc5d57fa700 1 -- 172.21.15.27:0/2932273231 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c801b630 con 0x7fc5d815dab0 2023-10-06T01:41:30.567 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.563+0000 7fc5d57fa700 1 -- 172.21.15.27:0/2932273231 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c801b790 con 0x7fc5d815dab0 2023-10-06T01:41:30.567 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.563+0000 7fc5df765700 1 -- 172.21.15.27:0/2932273231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5d8152c40 con 0x7fc5d815dab0 2023-10-06T01:41:30.567 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.563+0000 7fc5d57fa700 1 -- 172.21.15.27:0/2932273231 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c801bb30 con 0x7fc5d815dab0 2023-10-06T01:41:30.567 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.563+0000 7fc5df765700 1 -- 172.21.15.27:0/2932273231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5d8152610 con 0x7fc5d815dab0 2023-10-06T01:41:30.568 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.563+0000 7fc5df765700 1 -- 172.21.15.27:0/2932273231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5d8064050 con 0x7fc5d815dab0 2023-10-06T01:41:30.568 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.564+0000 7fc5d57fa700 1 -- 172.21.15.27:0/2932273231 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc5c801c400 con 0x7fc5d815dab0 2023-10-06T01:41:30.568 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.564+0000 7fc5d57fa700 1 --2- 172.21.15.27:0/2932273231 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc5c4066f30 0x7fc5c40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:30.568 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.564+0000 7fc5d57fa700 1 -- 172.21.15.27:0/2932273231 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fc5c80866c0 con 0x7fc5d815dab0 2023-10-06T01:41:30.568 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.565+0000 7fc5dd501700 1 --2- 172.21.15.27:0/2932273231 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc5c4066f30 0x7fc5c40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:30.569 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.565+0000 7fc5dd501700 1 --2- 172.21.15.27:0/2932273231 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc5c4066f30 0x7fc5c40693e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0007fd0 tx=0x7fc5d0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:30.570 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.569+0000 7fc5d57fa700 1 -- 172.21.15.27:0/2932273231 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc5c8053730 con 0x7fc5d815dab0 2023-10-06T01:41:30.764 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.763+0000 7fc5df765700 1 -- 172.21.15.27:0/2932273231 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc5d80689d0 con 0x7fc5d815dab0 2023-10-06T01:41:30.765 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.764+0000 7fc5d57fa700 1 -- 172.21.15.27:0/2932273231 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc5c8009880 con 0x7fc5d815dab0 2023-10-06T01:41:30.765 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2023-10-06T01:41:30.767 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.765+0000 7fc5df765700 1 -- 172.21.15.27:0/2932273231 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc5c4066f30 msgr2=0x7fc5c40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:30.767 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.766+0000 7fc5df765700 1 --2- 172.21.15.27:0/2932273231 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc5c4066f30 0x7fc5c40693e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc5d0007fd0 tx=0x7fc5d0009040 comp rx=0 tx=0).stop 2023-10-06T01:41:30.767 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.766+0000 7fc5df765700 1 -- 172.21.15.27:0/2932273231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5d815dab0 msgr2=0x7fc5d8069050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:30.767 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.766+0000 7fc5df765700 1 --2- 172.21.15.27:0/2932273231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5d815dab0 0x7fc5d8069050 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8088d00 tx=0x7fc5c8019000 comp rx=0 tx=0).stop 2023-10-06T01:41:30.768 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.766+0000 7fc5df765700 1 -- 172.21.15.27:0/2932273231 shutdown_connections 2023-10-06T01:41:30.768 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.766+0000 7fc5df765700 1 --2- 172.21.15.27:0/2932273231 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc5c4066f30 0x7fc5c40693e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:30.768 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.766+0000 7fc5df765700 1 --2- 172.21.15.27:0/2932273231 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc5d8159500 0x7fc5d80b43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:30.768 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.766+0000 7fc5df765700 1 --2- 172.21.15.27:0/2932273231 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc5d815dab0 0x7fc5d8069050 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:30.768 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.766+0000 7fc5df765700 1 --2- 172.21.15.27:0/2932273231 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc5d81598d0 0x7fc5d80b4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:30.768 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.766+0000 7fc5df765700 1 -- 172.21.15.27:0/2932273231 >> 172.21.15.27:0/2932273231 conn(0x7fc5d8067630 msgr2=0x7fc5d80ceb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:30.768 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.766+0000 7fc5df765700 1 -- 172.21.15.27:0/2932273231 shutdown_connections 2023-10-06T01:41:30.768 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.766+0000 7fc5df765700 1 -- 172.21.15.27:0/2932273231 wait complete. 2023-10-06T01:41:30.779 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T01:41:30.898 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.897+0000 7f6f3ecbe700 1 Processor -- start 2023-10-06T01:41:30.899 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.897+0000 7f6f3ecbe700 1 -- start start 2023-10-06T01:41:30.901 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.898+0000 7f6f3ecbe700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6f380cbe50 0x7f6f380c66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:30.901 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.898+0000 7f6f3ecbe700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f380ae140 con 0x7f6f380cdc00 2023-10-06T01:41:30.902 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.898+0000 7f6f3ecbe700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f380cc2a0 con 0x7f6f380c6bf0 2023-10-06T01:41:30.902 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.898+0000 7f6f3ecbe700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f380c7070 con 0x7f6f380cbe50 2023-10-06T01:41:30.902 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.898+0000 7f6f3ca5a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6f380cdc00 0x7f6f380d1030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57088/0 (socket says 172.21.15.27:57088) 2023-10-06T01:41:30.902 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.898+0000 7f6f3ca5a700 1 -- 172.21.15.27:0/198006094 learned_addr learned my addr 172.21.15.27:0/198006094 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:30.902 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.898+0000 7f6f37fff700 1 --2- 172.21.15.27:0/198006094 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6f380cbe50 0x7f6f380c66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:30.902 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.898+0000 7f6f377fe700 1 -- 172.21.15.27:0/198006094 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3008149622 0 0) 0x7f6f380ae140 con 0x7f6f380cdc00 2023-10-06T01:41:30.902 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.898+0000 7f6f377fe700 1 -- 172.21.15.27:0/198006094 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f20003570 con 0x7f6f380cdc00 2023-10-06T01:41:30.902 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.898+0000 7f6f377fe700 1 -- 172.21.15.27:0/198006094 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2369618606 0 0) 0x7f6f380cc2a0 con 0x7f6f380c6bf0 2023-10-06T01:41:30.902 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.898+0000 7f6f377fe700 1 -- 172.21.15.27:0/198006094 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f380ae140 con 0x7f6f380c6bf0 2023-10-06T01:41:30.902 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.898+0000 7f6f37fff700 1 -- 172.21.15.27:0/198006094 >> v1:172.21.15.124:6789/0 conn(0x7f6f380c6bf0 legacy=0x7f6f380ae5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:30.902 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.898+0000 7f6f37fff700 1 -- 172.21.15.27:0/198006094 >> v1:172.21.15.27:6789/0 conn(0x7f6f380cdc00 legacy=0x7f6f380d1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:30.903 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.898+0000 7f6f37fff700 1 -- 172.21.15.27:0/198006094 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f380ad530 con 0x7f6f380cbe50 2023-10-06T01:41:30.903 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.899+0000 7f6f37fff700 1 --2- 172.21.15.27:0/198006094 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6f380cbe50 0x7f6f380c66b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c009910 tx=0x7f6f2c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc7ec63a9c18f0b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:30.903 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.899+0000 7f6f377fe700 1 -- 172.21.15.27:0/198006094 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f2c011630 con 0x7f6f380cbe50 2023-10-06T01:41:30.903 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.899+0000 7f6f377fe700 1 -- 172.21.15.27:0/198006094 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f2c011790 con 0x7f6f380cbe50 2023-10-06T01:41:30.903 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.899+0000 7f6f377fe700 1 -- 172.21.15.27:0/198006094 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f2c011b30 con 0x7f6f380cbe50 2023-10-06T01:41:30.903 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.899+0000 7f6f3ecbe700 1 -- 172.21.15.27:0/198006094 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6f380cbe50 msgr2=0x7f6f380c66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:30.903 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.899+0000 7f6f3ecbe700 1 --2- 172.21.15.27:0/198006094 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6f380cbe50 0x7f6f380c66b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c009910 tx=0x7f6f2c0091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:30.903 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.899+0000 7f6f3ecbe700 1 -- 172.21.15.27:0/198006094 shutdown_connections 2023-10-06T01:41:30.903 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.899+0000 7f6f3ecbe700 1 --2- 172.21.15.27:0/198006094 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6f380cbe50 0x7f6f380c66b0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:30.903 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.899+0000 7f6f3ecbe700 1 -- 172.21.15.27:0/198006094 >> 172.21.15.27:0/198006094 conn(0x7f6f38099990 msgr2=0x7f6f380b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:30.904 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.899+0000 7f6f3ecbe700 1 -- 172.21.15.27:0/198006094 shutdown_connections 2023-10-06T01:41:30.904 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.899+0000 7f6f3ecbe700 1 -- 172.21.15.27:0/198006094 wait complete. 2023-10-06T01:41:30.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.900+0000 7f6f3ecbe700 1 Processor -- start 2023-10-06T01:41:30.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.901+0000 7f6f3ecbe700 1 -- start start 2023-10-06T01:41:30.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.901+0000 7f6f3ecbe700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6f380c6bf0 0x7f6f38157640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:30.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.901+0000 7f6f3ecbe700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6f380cbe50 0x7f6f380b1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:30.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.901+0000 7f6f3ecbe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f380cdc00 0x7f6f380b2500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:30.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.901+0000 7f6f3ecbe700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f380b13e0 con 0x7f6f380cdc00 2023-10-06T01:41:30.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.901+0000 7f6f3ecbe700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f38067db0 con 0x7f6f380c6bf0 2023-10-06T01:41:30.908 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.901+0000 7f6f3ecbe700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f380b1100 con 0x7f6f380cbe50 2023-10-06T01:41:30.908 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.901+0000 7f6f3d25b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f380cdc00 0x7f6f380b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:30.908 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.902+0000 7f6f3ca5a700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6f380c6bf0 0x7f6f38157640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:30.908 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.902+0000 7f6f3d25b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f380cdc00 0x7f6f380b2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52448/0 (socket says 172.21.15.27:52448) 2023-10-06T01:41:30.908 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.902+0000 7f6f3d25b700 1 -- 172.21.15.27:0/1975294344 learned_addr learned my addr 172.21.15.27:0/1975294344 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:30.908 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.902+0000 7f6f37fff700 1 --2- 172.21.15.27:0/1975294344 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6f380cbe50 0x7f6f380b1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:30.908 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.902+0000 7f6f3d25b700 1 -- 172.21.15.27:0/1975294344 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6f380cbe50 msgr2=0x7f6f380b1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:30.908 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.902+0000 7f6f3d25b700 1 --2- 172.21.15.27:0/1975294344 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6f380cbe50 0x7f6f380b1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:30.908 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.902+0000 7f6f3d25b700 1 -- 172.21.15.27:0/1975294344 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6f380c6bf0 msgr2=0x7f6f38157640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:30.909 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.902+0000 7f6f3d25b700 1 --2- 172.21.15.27:0/1975294344 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6f380c6bf0 0x7f6f38157640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:30.909 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.902+0000 7f6f3d25b700 1 -- 172.21.15.27:0/1975294344 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f2c009420 con 0x7f6f380cdc00 2023-10-06T01:41:30.909 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.902+0000 7f6f3d25b700 1 --2- 172.21.15.27:0/1975294344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f380cdc00 0x7f6f380b2500 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6f3000c910 tx=0x7f6f300027a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:30.909 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.902+0000 7f6f34ff9700 1 -- 172.21.15.27:0/1975294344 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f30012630 con 0x7f6f380cdc00 2023-10-06T01:41:30.909 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.902+0000 7f6f34ff9700 1 -- 172.21.15.27:0/1975294344 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f300127b0 con 0x7f6f380cdc00 2023-10-06T01:41:30.909 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.902+0000 7f6f3ecbe700 1 -- 172.21.15.27:0/1975294344 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f38067760 con 0x7f6f380cdc00 2023-10-06T01:41:30.909 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.902+0000 7f6f34ff9700 1 -- 172.21.15.27:0/1975294344 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f30012b70 con 0x7f6f380cdc00 2023-10-06T01:41:30.910 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.902+0000 7f6f3ecbe700 1 -- 172.21.15.27:0/1975294344 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f3816e140 con 0x7f6f380cdc00 2023-10-06T01:41:30.910 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.903+0000 7f6f3ecbe700 1 -- 172.21.15.27:0/1975294344 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f38067f20 con 0x7f6f380cdc00 2023-10-06T01:41:30.910 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.904+0000 7f6f34ff9700 1 -- 172.21.15.27:0/1975294344 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6f3001b400 con 0x7f6f380cdc00 2023-10-06T01:41:30.910 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.904+0000 7f6f34ff9700 1 --2- 172.21.15.27:0/1975294344 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6f24066f30 0x7f6f240693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:30.910 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.904+0000 7f6f34ff9700 1 -- 172.21.15.27:0/1975294344 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f6f300849a0 con 0x7f6f380cdc00 2023-10-06T01:41:30.910 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.904+0000 7f6f3ca5a700 1 --2- 172.21.15.27:0/1975294344 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6f24066f30 0x7f6f240693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:30.910 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.905+0000 7f6f3ca5a700 1 --2- 172.21.15.27:0/1975294344 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6f24066f30 0x7f6f240693e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6f28001230 tx=0x7f6f28009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:30.911 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:30.908+0000 7f6f34ff9700 1 -- 172.21.15.27:0/1975294344 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6f30051ab0 con 0x7f6f380cdc00 2023-10-06T01:41:31.145 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.144+0000 7f6f3ecbe700 1 -- 172.21.15.27:0/1975294344 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f38064050 con 0x7f6f380cdc00 2023-10-06T01:41:31.146 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.145+0000 7f6f34ff9700 1 -- 172.21.15.27:0/1975294344 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4594 (secure 0 0 0) 0x7f6f300550d0 con 0x7f6f380cdc00 2023-10-06T01:41:31.146 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:31.146 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4261,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6837/2699906167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6836","nonce":2699906167},{"type":"v1","addr":"172.21.15.124:6837","nonce":2699906167}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/313372271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":313372271},{"type":"v1","addr":"172.21.15.27:6834","nonce":313372271}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4287,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/4257743842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4257743842},{"type":"v1","addr":"172.21.15.27:6835","nonce":4257743842}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T01:41:12.735970+0000","modified":"2023-10-06T01:41:16.358924+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.124:6835/3803994994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":3803994994},{"type":"v1","addr":"172.21.15.124:6835","nonce":3803994994}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-06T01:41:31.149 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.147+0000 7f6f3ecbe700 1 -- 172.21.15.27:0/1975294344 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6f24066f30 msgr2=0x7f6f240693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:31.149 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.147+0000 7f6f3ecbe700 1 --2- 172.21.15.27:0/1975294344 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6f24066f30 0x7f6f240693e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6f28001230 tx=0x7f6f28009040 comp rx=0 tx=0).stop 2023-10-06T01:41:31.149 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.147+0000 7f6f3ecbe700 1 -- 172.21.15.27:0/1975294344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f380cdc00 msgr2=0x7f6f380b2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:31.149 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.147+0000 7f6f3ecbe700 1 --2- 172.21.15.27:0/1975294344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f380cdc00 0x7f6f380b2500 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6f3000c910 tx=0x7f6f300027a0 comp rx=0 tx=0).stop 2023-10-06T01:41:31.149 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.147+0000 7f6f3ecbe700 1 -- 172.21.15.27:0/1975294344 shutdown_connections 2023-10-06T01:41:31.149 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.147+0000 7f6f3ecbe700 1 --2- 172.21.15.27:0/1975294344 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6f24066f30 0x7f6f240693e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:31.149 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.147+0000 7f6f3ecbe700 1 --2- 172.21.15.27:0/1975294344 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6f380cdc00 0x7f6f380b2500 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:31.149 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.147+0000 7f6f3ecbe700 1 --2- 172.21.15.27:0/1975294344 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6f380c6bf0 0x7f6f38157640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:31.149 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.147+0000 7f6f3ecbe700 1 --2- 172.21.15.27:0/1975294344 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6f380cbe50 0x7f6f380b1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:31.149 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.147+0000 7f6f3ecbe700 1 -- 172.21.15.27:0/1975294344 >> 172.21.15.27:0/1975294344 conn(0x7f6f38099990 msgr2=0x7f6f380ae930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:31.150 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.147+0000 7f6f3ecbe700 1 -- 172.21.15.27:0/1975294344 shutdown_connections 2023-10-06T01:41:31.150 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.147+0000 7f6f3ecbe700 1 -- 172.21.15.27:0/1975294344 wait complete. 2023-10-06T01:41:31.150 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 7 2023-10-06T01:41:31.160 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T01:41:31.280 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.278+0000 7fcaf54a7700 1 Processor -- start 2023-10-06T01:41:31.280 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.278+0000 7fcaf54a7700 1 -- start start 2023-10-06T01:41:31.282 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.279+0000 7fcaf54a7700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcaf00af6b0 0x7fcaf0067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:31.282 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.279+0000 7fcaf54a7700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcaf01598e0 0x7fcaf0068400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:31.282 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.279+0000 7fcaf54a7700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf00adbb0 con 0x7fcaf0159510 2023-10-06T01:41:31.282 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.279+0000 7fcaf54a7700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcaf00ae210 con 0x7fcaf01598e0 2023-10-06T01:41:31.282 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.279+0000 7fcaf54a7700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf00ad5f0 con 0x7fcaf00af6b0 2023-10-06T01:41:31.282 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.279+0000 7fcaef7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcaf01598e0 0x7fcaf0068400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:31.283 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.279+0000 7fcaef7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fcaf01598e0 0x7fcaf0068400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52462/0 (socket says 172.21.15.27:52462) 2023-10-06T01:41:31.283 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.279+0000 7fcaef7fe700 1 -- 172.21.15.27:0/595899125 learned_addr learned my addr 172.21.15.27:0/595899125 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:31.283 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.280+0000 7fcaee7fc700 1 --2- 172.21.15.27:0/595899125 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcaf00af6b0 0x7fcaf0067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:31.283 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.280+0000 7fcaef7fe700 1 -- 172.21.15.27:0/595899125 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcaf00af6b0 msgr2=0x7fcaf0067ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:31.283 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.280+0000 7fcaef7fe700 1 --2- 172.21.15.27:0/595899125 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcaf00af6b0 0x7fcaf0067ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:31.283 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.280+0000 7fcaef7fe700 1 -- 172.21.15.27:0/595899125 >> v1:172.21.15.124:6789/0 conn(0x7fcaf0159510 legacy=0x7fcaf00acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:31.283 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.280+0000 7fcaef7fe700 1 -- 172.21.15.27:0/595899125 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf00ad8d0 con 0x7fcaf01598e0 2023-10-06T01:41:31.283 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.280+0000 7fcaee7fc700 1 --2- 172.21.15.27:0/595899125 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcaf00af6b0 0x7fcaf0067ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:31.283 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.280+0000 7fcaef7fe700 1 --2- 172.21.15.27:0/595899125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf01598e0 0x7fcaf0068400 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcae0009f20 tx=0x7fcae0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b36741a532736f8e server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:31.283 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.280+0000 7fcaedffb700 1 -- 172.21.15.27:0/595899125 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae000e070 con 0x7fcaf01598e0 2023-10-06T01:41:31.283 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.280+0000 7fcaedffb700 1 -- 172.21.15.27:0/595899125 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcae000ba80 con 0x7fcaf01598e0 2023-10-06T01:41:31.283 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.280+0000 7fcaedffb700 1 -- 172.21.15.27:0/595899125 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae00092e0 con 0x7fcaf01598e0 2023-10-06T01:41:31.284 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.280+0000 7fcaf54a7700 1 -- 172.21.15.27:0/595899125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf01598e0 msgr2=0x7fcaf0068400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:31.284 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.280+0000 7fcaf54a7700 1 --2- 172.21.15.27:0/595899125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf01598e0 0x7fcaf0068400 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcae0009f20 tx=0x7fcae0009840 comp rx=0 tx=0).stop 2023-10-06T01:41:31.284 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.281+0000 7fcaf54a7700 1 -- 172.21.15.27:0/595899125 shutdown_connections 2023-10-06T01:41:31.284 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.281+0000 7fcaf54a7700 1 --2- 172.21.15.27:0/595899125 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf01598e0 0x7fcaf0068400 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:31.284 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.281+0000 7fcaf54a7700 1 --2- 172.21.15.27:0/595899125 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcaf00af6b0 0x7fcaf0067ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:31.284 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.281+0000 7fcaf54a7700 1 -- 172.21.15.27:0/595899125 >> 172.21.15.27:0/595899125 conn(0x7fcaf006e6b0 msgr2=0x7fcaf006afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:31.284 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.281+0000 7fcaf54a7700 1 -- 172.21.15.27:0/595899125 shutdown_connections 2023-10-06T01:41:31.284 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.281+0000 7fcaf54a7700 1 -- 172.21.15.27:0/595899125 wait complete. 2023-10-06T01:41:31.287 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.282+0000 7fcaf54a7700 1 Processor -- start 2023-10-06T01:41:31.287 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.282+0000 7fcaf54a7700 1 -- start start 2023-10-06T01:41:31.287 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.282+0000 7fcaf54a7700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf00af6b0 0x7fcaf016a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:31.287 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.283+0000 7fcaf54a7700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fcaf0159510 0x7fcaf0153be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:31.287 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.283+0000 7fcaf54a7700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcaf01598e0 0x7fcaf0169ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:31.287 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.283+0000 7fcaf54a7700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf0154120 con 0x7fcaf00af6b0 2023-10-06T01:41:31.287 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.283+0000 7fcaf54a7700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf016a400 con 0x7fcaf0159510 2023-10-06T01:41:31.287 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.283+0000 7fcaf54a7700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf0154590 con 0x7fcaf01598e0 2023-10-06T01:41:31.287 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.283+0000 7fcaeeffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf00af6b0 0x7fcaf016a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:31.287 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.283+0000 7fcaeeffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf00af6b0 0x7fcaf016a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52466/0 (socket says 172.21.15.27:52466) 2023-10-06T01:41:31.287 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.283+0000 7fcaeeffd700 1 -- 172.21.15.27:0/3833924372 learned_addr learned my addr 172.21.15.27:0/3833924372 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:31.288 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.283+0000 7fcaee7fc700 1 --2- 172.21.15.27:0/3833924372 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fcaf0159510 0x7fcaf0153be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:31.288 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.283+0000 7fcaef7fe700 1 --2- 172.21.15.27:0/3833924372 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcaf01598e0 0x7fcaf0169ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:31.288 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.283+0000 7fcaeeffd700 1 -- 172.21.15.27:0/3833924372 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcaf01598e0 msgr2=0x7fcaf0169ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:31.288 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.283+0000 7fcaeeffd700 1 --2- 172.21.15.27:0/3833924372 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcaf01598e0 0x7fcaf0169ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:31.288 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.283+0000 7fcaeeffd700 1 -- 172.21.15.27:0/3833924372 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fcaf0159510 msgr2=0x7fcaf0153be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:31.288 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.283+0000 7fcaeeffd700 1 --2- 172.21.15.27:0/3833924372 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fcaf0159510 0x7fcaf0153be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:31.288 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.283+0000 7fcaeeffd700 1 -- 172.21.15.27:0/3833924372 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae0009ae0 con 0x7fcaf00af6b0 2023-10-06T01:41:31.288 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.283+0000 7fcaee7fc700 1 --2- 172.21.15.27:0/3833924372 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fcaf0159510 0x7fcaf0153be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:31.288 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.283+0000 7fcaeeffd700 1 --2- 172.21.15.27:0/3833924372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf00af6b0 0x7fcaf016a620 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fcae800c5c0 tx=0x7fcae800c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:31.288 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.283+0000 7fcae6ffd700 1 -- 172.21.15.27:0/3833924372 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae8011590 con 0x7fcaf00af6b0 2023-10-06T01:41:31.288 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.283+0000 7fcae6ffd700 1 -- 172.21.15.27:0/3833924372 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcae8011710 con 0x7fcaf00af6b0 2023-10-06T01:41:31.288 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.283+0000 7fcaf54a7700 1 -- 172.21.15.27:0/3833924372 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaf01573b0 con 0x7fcaf00af6b0 2023-10-06T01:41:31.289 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.283+0000 7fcae6ffd700 1 -- 172.21.15.27:0/3833924372 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae8011ad0 con 0x7fcaf00af6b0 2023-10-06T01:41:31.289 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.283+0000 7fcaf54a7700 1 -- 172.21.15.27:0/3833924372 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaf0157c50 con 0x7fcaf00af6b0 2023-10-06T01:41:31.289 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.284+0000 7fcaf54a7700 1 -- 172.21.15.27:0/3833924372 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaf0064050 con 0x7fcaf00af6b0 2023-10-06T01:41:31.289 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.285+0000 7fcae6ffd700 1 -- 172.21.15.27:0/3833924372 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fcae8012420 con 0x7fcaf00af6b0 2023-10-06T01:41:31.289 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.285+0000 7fcae6ffd700 1 --2- 172.21.15.27:0/3833924372 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fcad0066f30 0x7fcad00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:31.289 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.285+0000 7fcae6ffd700 1 -- 172.21.15.27:0/3833924372 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fcae8083a20 con 0x7fcaf00af6b0 2023-10-06T01:41:31.289 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.285+0000 7fcaee7fc700 1 --2- 172.21.15.27:0/3833924372 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fcad0066f30 0x7fcad00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:31.289 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.286+0000 7fcaee7fc700 1 --2- 172.21.15.27:0/3833924372 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fcad0066f30 0x7fcad00693e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcad8006f00 tx=0x7fcad8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:31.290 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.289+0000 7fcae6ffd700 1 -- 172.21.15.27:0/3833924372 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fcae804b5a0 con 0x7fcaf00af6b0 2023-10-06T01:41:31.490 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.488+0000 7fcaf54a7700 1 -- 172.21.15.27:0/3833924372 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcaf00adea0 con 0x7fcaf00af6b0 2023-10-06T01:41:31.491 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.489+0000 7fcae6ffd700 1 -- 172.21.15.27:0/3833924372 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16866 (secure 0 0 0) 0x7fcae8017070 con 0x7fcaf00af6b0 2023-10-06T01:41:31.491 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:31.492 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":20,"fsid":"20a7bff0-74f8-4f2f-b6c6-899e804bc965","created":"2023-10-06T01:40:54.474215+0000","modified":"2023-10-06T01:41:14.324546+0000","last_up_change":"2023-10-06T01:41:05.242847+0000","last_in_change":"2023-10-06T01:40:56.890960+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-10-06T01:41:05.685386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-10-06T01:41:06.092995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T01:41:10.434257+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T01:41:11.700262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b902c86d-800c-40c5-b152-b545e50d02c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":90644},{"type":"v1","addr":"172.21.15.27:6801","nonce":90644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":90644},{"type":"v1","addr":"172.21.15.27:6803","nonce":90644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":90644},{"type":"v1","addr":"172.21.15.27:6807","nonce":90644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":90644},{"type":"v1","addr":"172.21.15.27:6805","nonce":90644}]},"public_addr":"172.21.15.27:6801/90644","cluster_addr":"172.21.15.27:6803/90644","heartbeat_back_addr":"172.21.15.27:6807/90644","heartbeat_front_addr":"172.21.15.27:6805/90644","state":["exists","up"]},{"osd":1,"uuid":"5b4a70e5-b06e-48cd-bfc4-7962c5d05386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":90643},{"type":"v1","addr":"172.21.15.27:6809","nonce":90643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":90643},{"type":"v1","addr":"172.21.15.27:6811","nonce":90643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":90643},{"type":"v1","addr":"172.21.15.27:6815","nonce":90643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":90643},{"type":"v1","addr":"172.21.15.27:6813","nonce":90643}]},"public_addr":"172.21.15.27:6809/90643","cluster_addr":"172.21.15.27:6811/90643","heartbeat_back_addr":"172.21.15.27:6815/90643","heartbeat_front_addr":"172.21.15.27:6813/90643","state":["exists","up"]},{"osd":2,"uuid":"22d9de46-4073-4336-9122-5227e601eca0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":90645},{"type":"v1","addr":"172.21.15.27:6817","nonce":90645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":90645},{"type":"v1","addr":"172.21.15.27:6819","nonce":90645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":90645},{"type":"v1","addr":"172.21.15.27:6823","nonce":90645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":90645},{"type":"v1","addr":"172.21.15.27:6821","nonce":90645}]},"public_addr":"172.21.15.27:6817/90645","cluster_addr":"172.21.15.27:6819/90645","heartbeat_back_addr":"172.21.15.27:6823/90645","heartbeat_front_addr":"172.21.15.27:6821/90645","state":["exists","up"]},{"osd":3,"uuid":"438bb2d3-34e2-4b7c-9734-8360c1b67650","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":90668},{"type":"v1","addr":"172.21.15.27:6825","nonce":90668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":90668},{"type":"v1","addr":"172.21.15.27:6827","nonce":90668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":90668},{"type":"v1","addr":"172.21.15.27:6831","nonce":90668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":90668},{"type":"v1","addr":"172.21.15.27:6829","nonce":90668}]},"public_addr":"172.21.15.27:6825/90668","cluster_addr":"172.21.15.27:6827/90668","heartbeat_back_addr":"172.21.15.27:6831/90668","heartbeat_front_addr":"172.21.15.27:6829/90668","state":["exists","up"]},{"osd":4,"uuid":"e1956a8e-e0f8-4e7c-acbb-40037415a354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":90863},{"type":"v1","addr":"172.21.15.124:6801","nonce":90863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":90863},{"type":"v1","addr":"172.21.15.124:6803","nonce":90863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":90863},{"type":"v1","addr":"172.21.15.124:6807","nonce":90863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":90863},{"type":"v1","addr":"172.21.15.124:6805","nonce":90863}]},"public_addr":"172.21.15.124:6801/90863","cluster_addr":"172.21.15.124:6803/90863","heartbeat_back_addr":"172.21.15.124:6807/90863","heartbeat_front_addr":"172.21.15.124:6805/90863","state":["exists","up"]},{"osd":5,"uuid":"306c59d4-846d-4b73-b1c9-051bec9e9368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":90885},{"type":"v1","addr":"172.21.15.124:6817","nonce":90885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":90885},{"type":"v1","addr":"172.21.15.124:6819","nonce":90885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":90885},{"type":"v1","addr":"172.21.15.124:6823","nonce":90885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":90885},{"type":"v1","addr":"172.21.15.124:6821","nonce":90885}]},"public_addr":"172.21.15.124:6817/90885","cluster_addr":"172.21.15.124:6819/90885","heartbeat_back_addr":"172.21.15.124:6823/90885","heartbeat_front_addr":"172.21.15.124:6821/90885","state":["exists","up"]},{"osd":6,"uuid":"15463e53-ac8d-47d1-b4f6-c0d31f29e4a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":90864},{"type":"v1","addr":"172.21.15.124:6809","nonce":90864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":90864},{"type":"v1","addr":"172.21.15.124:6811","nonce":90864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":90864},{"type":"v1","addr":"172.21.15.124:6815","nonce":90864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":90864},{"type":"v1","addr":"172.21.15.124:6813","nonce":90864}]},"public_addr":"172.21.15.124:6809/90864","cluster_addr":"172.21.15.124:6811/90864","heartbeat_back_addr":"172.21.15.124:6815/90864","heartbeat_front_addr":"172.21.15.124:6813/90864","state":["exists","up"]},{"osd":7,"uuid":"ae463a90-a9d3-498f-86fa-47a35b3fe889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":90894},{"type":"v1","addr":"172.21.15.124:6825","nonce":90894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":90894},{"type":"v1","addr":"172.21.15.124:6827","nonce":90894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":90894},{"type":"v1","addr":"172.21.15.124:6831","nonce":90894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":90894},{"type":"v1","addr":"172.21.15.124:6829","nonce":90894}]},"public_addr":"172.21.15.124:6825/90894","cluster_addr":"172.21.15.124:6827/90894","heartbeat_back_addr":"172.21.15.124:6831/90894","heartbeat_front_addr":"172.21.15.124:6829/90894","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.827704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.839951+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.032230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.050605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.987908+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.031290+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.080707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.053692+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T01:41:31.493 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.491+0000 7fcaf54a7700 1 -- 172.21.15.27:0/3833924372 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fcad0066f30 msgr2=0x7fcad00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:31.493 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.491+0000 7fcaf54a7700 1 --2- 172.21.15.27:0/3833924372 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fcad0066f30 0x7fcad00693e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcad8006f00 tx=0x7fcad8008040 comp rx=0 tx=0).stop 2023-10-06T01:41:31.494 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.491+0000 7fcaf54a7700 1 -- 172.21.15.27:0/3833924372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf00af6b0 msgr2=0x7fcaf016a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:31.494 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.491+0000 7fcaf54a7700 1 --2- 172.21.15.27:0/3833924372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf00af6b0 0x7fcaf016a620 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fcae800c5c0 tx=0x7fcae800c980 comp rx=0 tx=0).stop 2023-10-06T01:41:31.494 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.491+0000 7fcaf54a7700 1 -- 172.21.15.27:0/3833924372 shutdown_connections 2023-10-06T01:41:31.494 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.491+0000 7fcaf54a7700 1 --2- 172.21.15.27:0/3833924372 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fcad0066f30 0x7fcad00693e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:31.494 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.491+0000 7fcaf54a7700 1 --2- 172.21.15.27:0/3833924372 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fcaf01598e0 0x7fcaf0169ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:31.495 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.491+0000 7fcaf54a7700 1 --2- 172.21.15.27:0/3833924372 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fcaf00af6b0 0x7fcaf016a620 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:31.495 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.491+0000 7fcaf54a7700 1 --2- 172.21.15.27:0/3833924372 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fcaf0159510 0x7fcaf0153be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:31.495 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.491+0000 7fcaf54a7700 1 -- 172.21.15.27:0/3833924372 >> 172.21.15.27:0/3833924372 conn(0x7fcaf006e6b0 msgr2=0x7fcaf0154ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:31.495 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.491+0000 7fcaf54a7700 1 -- 172.21.15.27:0/3833924372 shutdown_connections 2023-10-06T01:41:31.495 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.491+0000 7fcaf54a7700 1 -- 172.21.15.27:0/3833924372 wait complete. 2023-10-06T01:41:31.505 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T01:41:31.624 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.623+0000 7f51d2df9700 1 Processor -- start 2023-10-06T01:41:31.624 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.623+0000 7f51d2df9700 1 -- start start 2023-10-06T01:41:31.628 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.624+0000 7f51d2df9700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f51cc0b0310 0x7f51cc0b0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:31.628 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.624+0000 7f51d2df9700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51cc069f70 con 0x7f51cc15d480 2023-10-06T01:41:31.628 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.624+0000 7f51d2df9700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51cc0b0c90 con 0x7f51cc15d850 2023-10-06T01:41:31.628 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.624+0000 7f51d2df9700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51cc0687c0 con 0x7f51cc0b0310 2023-10-06T01:41:31.628 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.624+0000 7f51d1df7700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f51cc15d480 0x7f51cc069860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57104/0 (socket says 172.21.15.27:57104) 2023-10-06T01:41:31.628 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.624+0000 7f51d1df7700 1 -- 172.21.15.27:0/4106970267 learned_addr learned my addr 172.21.15.27:0/4106970267 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:31.628 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.624+0000 7f51d25f8700 1 --2- 172.21.15.27:0/4106970267 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f51cc0b0310 0x7f51cc0b0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:31.628 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.624+0000 7f51d0df5700 1 -- 172.21.15.27:0/4106970267 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2991314370 0 0) 0x7f51cc069f70 con 0x7f51cc15d480 2023-10-06T01:41:31.629 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.624+0000 7f51d0df5700 1 -- 172.21.15.27:0/4106970267 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51b8003570 con 0x7f51cc15d480 2023-10-06T01:41:31.629 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.624+0000 7f51d0df5700 1 -- 172.21.15.27:0/4106970267 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2172184733 0 0) 0x7f51cc0b0c90 con 0x7f51cc15d850 2023-10-06T01:41:31.629 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.624+0000 7f51d0df5700 1 -- 172.21.15.27:0/4106970267 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51cc069f70 con 0x7f51cc15d850 2023-10-06T01:41:31.629 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.625+0000 7f51d0df5700 1 -- 172.21.15.27:0/4106970267 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1589605514 0 0) 0x7f51b8003570 con 0x7f51cc15d480 2023-10-06T01:41:31.629 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.625+0000 7f51d0df5700 1 -- 172.21.15.27:0/4106970267 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f51cc0b0310 msgr2=0x7f51cc0b0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:31.629 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.625+0000 7f51d0df5700 1 --2- 172.21.15.27:0/4106970267 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f51cc0b0310 0x7f51cc0b0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:31.629 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.625+0000 7f51d0df5700 1 -- 172.21.15.27:0/4106970267 >> v1:172.21.15.124:6789/0 conn(0x7f51cc15d850 legacy=0x7f51cc0b5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:31.629 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.625+0000 7f51d0df5700 1 -- 172.21.15.27:0/4106970267 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51cc068aa0 con 0x7f51cc15d480 2023-10-06T01:41:31.629 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.625+0000 7f51d0df5700 1 -- 172.21.15.27:0/4106970267 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2619078944 0 0) 0x7f51c40017f0 con 0x7f51cc15d480 2023-10-06T01:41:31.629 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.625+0000 7f51d0df5700 1 -- 172.21.15.27:0/4106970267 >> v1:172.21.15.27:6789/0 conn(0x7f51cc15d480 legacy=0x7f51cc069860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:31.629 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.625+0000 7f51d0df5700 1 --2- 172.21.15.27:0/4106970267 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f51b8003c20 0x7f51b8004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:31.630 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.625+0000 7f51d0df5700 1 --2- 172.21.15.27:0/4106970267 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f51cc0b0310 0x7f51b8004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:31.630 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.625+0000 7f51d0df5700 1 --2- 172.21.15.27:0/4106970267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51b8004d50 0x7f51b80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:31.630 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.625+0000 7f51d0df5700 1 -- 172.21.15.27:0/4106970267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51cc0687c0 con 0x7f51b8004d50 2023-10-06T01:41:31.630 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.625+0000 7f51d0df5700 1 -- 172.21.15.27:0/4106970267 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51b800bb70 con 0x7f51b8003c20 2023-10-06T01:41:31.630 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.625+0000 7f51d0df5700 1 -- 172.21.15.27:0/4106970267 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51b800bd30 con 0x7f51cc0b0310 2023-10-06T01:41:31.630 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.625+0000 7f51d25f8700 1 --2- 172.21.15.27:0/4106970267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51b8004d50 0x7f51b80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:31.630 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.625+0000 7f51d1df7700 1 --2- 172.21.15.27:0/4106970267 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f51b8003c20 0x7f51b8004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:31.630 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.625+0000 7f51d15f6700 1 --2- 172.21.15.27:0/4106970267 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f51cc0b0310 0x7f51b8004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:31.630 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.625+0000 7f51d25f8700 1 -- 172.21.15.27:0/4106970267 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f51cc0b0310 msgr2=0x7f51b8004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:31.630 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.625+0000 7f51d25f8700 1 --2- 172.21.15.27:0/4106970267 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f51cc0b0310 0x7f51b8004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:31.630 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.625+0000 7f51d25f8700 1 -- 172.21.15.27:0/4106970267 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f51b8003c20 msgr2=0x7f51b8004010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:31.631 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.625+0000 7f51d25f8700 1 --2- 172.21.15.27:0/4106970267 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f51b8003c20 0x7f51b8004010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:31.631 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.625+0000 7f51d25f8700 1 -- 172.21.15.27:0/4106970267 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51b8003570 con 0x7f51b8004d50 2023-10-06T01:41:31.631 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.625+0000 7f51d25f8700 1 --2- 172.21.15.27:0/4106970267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51b8004d50 0x7f51b80051c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f51c8009c00 tx=0x7f51c80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f9a996cff33e50c server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:31.631 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.625+0000 7f51d0df5700 1 -- 172.21.15.27:0/4106970267 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c800e710 con 0x7f51b8004d50 2023-10-06T01:41:31.631 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.625+0000 7f51d0df5700 1 -- 172.21.15.27:0/4106970267 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51c800e870 con 0x7f51b8004d50 2023-10-06T01:41:31.631 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.625+0000 7f51d0df5700 1 -- 172.21.15.27:0/4106970267 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c800ec30 con 0x7f51b8004d50 2023-10-06T01:41:31.631 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.626+0000 7f51d2df9700 1 -- 172.21.15.27:0/4106970267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51b8004d50 msgr2=0x7f51b80051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:31.631 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.626+0000 7f51d2df9700 1 --2- 172.21.15.27:0/4106970267 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51b8004d50 0x7f51b80051c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f51c8009c00 tx=0x7f51c80091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:31.631 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.626+0000 7f51d25f8700 1 -- 172.21.15.27:0/4106970267 reap_dead start 2023-10-06T01:41:31.631 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.626+0000 7f51d2df9700 1 -- 172.21.15.27:0/4106970267 shutdown_connections 2023-10-06T01:41:31.631 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.626+0000 7f51d2df9700 1 -- 172.21.15.27:0/4106970267 >> 172.21.15.27:0/4106970267 conn(0x7f51cc0ccf80 msgr2=0x7f51cc0cd380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:31.632 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.626+0000 7f51d2df9700 1 -- 172.21.15.27:0/4106970267 shutdown_connections 2023-10-06T01:41:31.632 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.626+0000 7f51d2df9700 1 -- 172.21.15.27:0/4106970267 wait complete. 2023-10-06T01:41:31.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.627+0000 7f51d2df9700 1 Processor -- start 2023-10-06T01:41:31.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.627+0000 7f51d2df9700 1 -- start start 2023-10-06T01:41:31.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.628+0000 7f51d2df9700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51b8004d50 0x7f51cc16da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:31.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.628+0000 7f51d2df9700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f51cc0ce310 0x7f51cc0adfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:31.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.628+0000 7f51d2df9700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f51cc0ce780 0x7f51cc0ae510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:31.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.628+0000 7f51d2df9700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51cc0a6df0 con 0x7f51b8004d50 2023-10-06T01:41:31.635 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.628+0000 7f51d2df9700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51cc16cf60 con 0x7f51cc0ce780 2023-10-06T01:41:31.635 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.628+0000 7f51d2df9700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51cc16aaa0 con 0x7f51cc0ce310 2023-10-06T01:41:31.635 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.628+0000 7f51d1df7700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51b8004d50 0x7f51cc16da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:31.635 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.628+0000 7f51d1df7700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51b8004d50 0x7f51cc16da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52472/0 (socket says 172.21.15.27:52472) 2023-10-06T01:41:31.635 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.628+0000 7f51d1df7700 1 -- 172.21.15.27:0/3762974369 learned_addr learned my addr 172.21.15.27:0/3762974369 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:31.635 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.628+0000 7f51d25f8700 1 --2- 172.21.15.27:0/3762974369 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f51cc0ce780 0x7f51cc0ae510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:31.635 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.628+0000 7f51d15f6700 1 --2- 172.21.15.27:0/3762974369 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f51cc0ce310 0x7f51cc0adfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:31.635 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.628+0000 7f51d1df7700 1 -- 172.21.15.27:0/3762974369 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f51cc0ce310 msgr2=0x7f51cc0adfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:31.635 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.628+0000 7f51d1df7700 1 --2- 172.21.15.27:0/3762974369 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f51cc0ce310 0x7f51cc0adfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:31.635 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.628+0000 7f51d1df7700 1 -- 172.21.15.27:0/3762974369 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f51cc0ce780 msgr2=0x7f51cc0ae510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:31.635 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.628+0000 7f51d1df7700 1 --2- 172.21.15.27:0/3762974369 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f51cc0ce780 0x7f51cc0ae510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:31.635 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.628+0000 7f51d1df7700 1 -- 172.21.15.27:0/3762974369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51cc169e90 con 0x7f51b8004d50 2023-10-06T01:41:31.636 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.628+0000 7f51d1df7700 1 --2- 172.21.15.27:0/3762974369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51b8004d50 0x7f51cc16da90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f51c4007a90 tx=0x7f51c4002190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:31.636 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.629+0000 7f51c1ffb700 1 -- 172.21.15.27:0/3762974369 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c4007ac0 con 0x7f51b8004d50 2023-10-06T01:41:31.636 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.629+0000 7f51c1ffb700 1 -- 172.21.15.27:0/3762974369 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51c400acc0 con 0x7f51b8004d50 2023-10-06T01:41:31.636 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.629+0000 7f51c1ffb700 1 -- 172.21.15.27:0/3762974369 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c4013640 con 0x7f51b8004d50 2023-10-06T01:41:31.636 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.629+0000 7f51d2df9700 1 -- 172.21.15.27:0/3762974369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51cc16a170 con 0x7f51b8004d50 2023-10-06T01:41:31.636 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.629+0000 7f51d2df9700 1 -- 172.21.15.27:0/3762974369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51cc16a7b0 con 0x7f51b8004d50 2023-10-06T01:41:31.636 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.630+0000 7f51d2df9700 1 -- 172.21.15.27:0/3762974369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51cc0ced40 con 0x7f51b8004d50 2023-10-06T01:41:31.636 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.630+0000 7f51c1ffb700 1 -- 172.21.15.27:0/3762974369 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f51c4013a30 con 0x7f51b8004d50 2023-10-06T01:41:31.636 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.630+0000 7f51c1ffb700 1 --2- 172.21.15.27:0/3762974369 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f51b0066f30 0x7f51b00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:31.636 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.631+0000 7f51c1ffb700 1 -- 172.21.15.27:0/3762974369 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f51c4085230 con 0x7f51b8004d50 2023-10-06T01:41:31.636 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.631+0000 7f51d15f6700 1 --2- 172.21.15.27:0/3762974369 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f51b0066f30 0x7f51b00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:31.636 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.631+0000 7f51d15f6700 1 --2- 172.21.15.27:0/3762974369 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f51b0066f30 0x7f51b00693e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f51bc007e40 tx=0x7f51bc007d30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:31.637 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.635+0000 7f51c1ffb700 1 -- 172.21.15.27:0/3762974369 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f51c4052340 con 0x7f51b8004d50 2023-10-06T01:41:31.872 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.871+0000 7f51d2df9700 1 -- 172.21.15.27:0/3762974369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51cc04c3c0 con 0x7f51b8004d50 2023-10-06T01:41:31.872 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.871+0000 7f51c1ffb700 1 -- 172.21.15.27:0/3762974369 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4594 (secure 0 0 0) 0x7f51c4055960 con 0x7f51b8004d50 2023-10-06T01:41:31.872 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:31.873 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4261,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6837/2699906167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6836","nonce":2699906167},{"type":"v1","addr":"172.21.15.124:6837","nonce":2699906167}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/313372271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":313372271},{"type":"v1","addr":"172.21.15.27:6834","nonce":313372271}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4287,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/4257743842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4257743842},{"type":"v1","addr":"172.21.15.27:6835","nonce":4257743842}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T01:41:12.735970+0000","modified":"2023-10-06T01:41:16.358924+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.124:6835/3803994994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":3803994994},{"type":"v1","addr":"172.21.15.124:6835","nonce":3803994994}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-06T01:41:31.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.873+0000 7f51d2df9700 1 -- 172.21.15.27:0/3762974369 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f51b0066f30 msgr2=0x7f51b00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:31.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.873+0000 7f51d2df9700 1 --2- 172.21.15.27:0/3762974369 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f51b0066f30 0x7f51b00693e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f51bc007e40 tx=0x7f51bc007d30 comp rx=0 tx=0).stop 2023-10-06T01:41:31.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.873+0000 7f51d2df9700 1 -- 172.21.15.27:0/3762974369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51b8004d50 msgr2=0x7f51cc16da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:31.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.873+0000 7f51d2df9700 1 --2- 172.21.15.27:0/3762974369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51b8004d50 0x7f51cc16da90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f51c4007a90 tx=0x7f51c4002190 comp rx=0 tx=0).stop 2023-10-06T01:41:31.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.873+0000 7f51d2df9700 1 -- 172.21.15.27:0/3762974369 shutdown_connections 2023-10-06T01:41:31.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.873+0000 7f51d2df9700 1 --2- 172.21.15.27:0/3762974369 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f51b0066f30 0x7f51b00693e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:31.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.873+0000 7f51d2df9700 1 --2- 172.21.15.27:0/3762974369 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f51cc0ce780 0x7f51cc0ae510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:31.876 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.873+0000 7f51d2df9700 1 --2- 172.21.15.27:0/3762974369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f51b8004d50 0x7f51cc16da90 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:31.876 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.873+0000 7f51d2df9700 1 --2- 172.21.15.27:0/3762974369 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f51cc0ce310 0x7f51cc0adfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:31.876 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.873+0000 7f51d2df9700 1 -- 172.21.15.27:0/3762974369 >> 172.21.15.27:0/3762974369 conn(0x7f51cc0ccf80 msgr2=0x7f51cc141d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:31.876 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.873+0000 7f51d2df9700 1 -- 172.21.15.27:0/3762974369 shutdown_connections 2023-10-06T01:41:31.876 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:31.873+0000 7f51d2df9700 1 -- 172.21.15.27:0/3762974369 wait complete. 2023-10-06T01:41:31.876 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 7 2023-10-06T01:41:32.289 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-06T01:41:12.735970+0000', 'modified': '2023-10-06T01:41:16.358924+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4305}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4305': {'gid': 4305, 'name': 'd', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.124:6835/3803994994', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.124:6834', 'nonce': 3803994994}, {'type': 'v1', 'addr': '172.21.15.124:6835', 'nonce': 3803994994}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-10-06T01:41:32.289 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-06T01:41:32.289 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-06T01:41:32.412 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.410+0000 7fa1ef6da700 1 Processor -- start 2023-10-06T01:41:32.412 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.411+0000 7fa1ef6da700 1 -- start start 2023-10-06T01:41:32.415 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.411+0000 7fa1ef6da700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa1e81598d0 0x7fa1e815d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:32.415 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.411+0000 7fa1ef6da700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fa1e815d8e0 0x7fa1e80cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:32.415 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.411+0000 7fa1ef6da700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1e815dd20 con 0x7fa1e8159500 2023-10-06T01:41:32.415 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.411+0000 7fa1ef6da700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1e80b1110 con 0x7fa1e815d8e0 2023-10-06T01:41:32.415 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.411+0000 7fa1ef6da700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e80b13b0 con 0x7fa1e81598d0 2023-10-06T01:41:32.415 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.412+0000 7fa1ecc75700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa1e81598d0 0x7fa1e815d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:32.415 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.412+0000 7fa1edc77700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fa1e815d8e0 0x7fa1e80cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:32.415 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.412+0000 7fa1ecc75700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa1e81598d0 0x7fa1e815d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.27:44926/0 (socket says 172.21.15.27:44926) 2023-10-06T01:41:32.415 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.412+0000 7fa1edc77700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fa1e815d8e0 0x7fa1e80cf6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.27:37716/0 (socket says 172.21.15.27:37716) 2023-10-06T01:41:32.415 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.412+0000 7fa1ecc75700 1 -- 172.21.15.27:0/835347119 learned_addr learned my addr 172.21.15.27:0/835347119 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:32.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.412+0000 7fa1ed476700 1 --1- 172.21.15.27:0/835347119 >> v1:172.21.15.124:6789/0 conn(0x7fa1e8159500 0x7fa1e80b3fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.27:50990/0 (socket says 172.21.15.27:50990) 2023-10-06T01:41:32.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.412+0000 7fa1edc77700 1 -- 172.21.15.27:0/835347119 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa1e81598d0 msgr2=0x7fa1e815d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:32.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.412+0000 7fa1edc77700 1 --2- 172.21.15.27:0/835347119 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa1e81598d0 0x7fa1e815d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:32.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.412+0000 7fa1edc77700 1 -- 172.21.15.27:0/835347119 >> v1:172.21.15.124:6789/0 conn(0x7fa1e8159500 legacy=0x7fa1e80b3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:32.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.412+0000 7fa1e7fff700 1 -- 172.21.15.27:0/835347119 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3872842802 0 0) 0x7fa1e815dd20 con 0x7fa1e8159500 2023-10-06T01:41:32.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.412+0000 7fa1edc77700 1 -- 172.21.15.27:0/835347119 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e80b1690 con 0x7fa1e815d8e0 2023-10-06T01:41:32.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.412+0000 7fa1ecc75700 1 --2- 172.21.15.27:0/835347119 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa1e81598d0 0x7fa1e815d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:41:32.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.412+0000 7fa1edc77700 1 --2- 172.21.15.27:0/835347119 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa1e815d8e0 0x7fa1e80cf6c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa1d8009e60 tx=0x7fa1d8009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=653378e397bf0c50 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:32.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.413+0000 7fa1e7fff700 1 -- 172.21.15.27:0/835347119 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d800e070 con 0x7fa1e815d8e0 2023-10-06T01:41:32.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.413+0000 7fa1e7fff700 1 -- 172.21.15.27:0/835347119 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1d800bb60 con 0x7fa1e815d8e0 2023-10-06T01:41:32.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.413+0000 7fa1e7fff700 1 -- 172.21.15.27:0/835347119 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d8012430 con 0x7fa1e815d8e0 2023-10-06T01:41:32.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.413+0000 7fa1ef6da700 1 -- 172.21.15.27:0/835347119 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa1e815d8e0 msgr2=0x7fa1e80cf6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:32.417 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.413+0000 7fa1ef6da700 1 --2- 172.21.15.27:0/835347119 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa1e815d8e0 0x7fa1e80cf6c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa1d8009e60 tx=0x7fa1d8009780 comp rx=0 tx=0).stop 2023-10-06T01:41:32.417 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.413+0000 7fa1ef6da700 1 -- 172.21.15.27:0/835347119 shutdown_connections 2023-10-06T01:41:32.417 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.413+0000 7fa1ef6da700 1 --2- 172.21.15.27:0/835347119 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa1e815d8e0 0x7fa1e80cf6c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:32.417 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.413+0000 7fa1ef6da700 1 --2- 172.21.15.27:0/835347119 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa1e81598d0 0x7fa1e815d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:32.417 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.413+0000 7fa1ef6da700 1 -- 172.21.15.27:0/835347119 >> 172.21.15.27:0/835347119 conn(0x7fa1e8067630 msgr2=0x7fa1e8069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:32.417 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.413+0000 7fa1ef6da700 1 -- 172.21.15.27:0/835347119 shutdown_connections 2023-10-06T01:41:32.417 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.413+0000 7fa1ef6da700 1 -- 172.21.15.27:0/835347119 wait complete. 2023-10-06T01:41:32.419 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.414+0000 7fa1ef6da700 1 Processor -- start 2023-10-06T01:41:32.419 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.414+0000 7fa1ef6da700 1 -- start start 2023-10-06T01:41:32.420 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.415+0000 7fa1ef6da700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa1e8159500 0x7fa1e814c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:32.420 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.415+0000 7fa1ef6da700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1e81598d0 0x7fa1e815b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:32.420 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.415+0000 7fa1ef6da700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa1e815d8e0 0x7fa1e815f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:32.420 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.415+0000 7fa1ef6da700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e814f5f0 con 0x7fa1e81598d0 2023-10-06T01:41:32.420 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.415+0000 7fa1ef6da700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e814e9e0 con 0x7fa1e8159500 2023-10-06T01:41:32.420 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.415+0000 7fa1ef6da700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e814ecc0 con 0x7fa1e815d8e0 2023-10-06T01:41:32.420 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.415+0000 7fa1ecc75700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1e81598d0 0x7fa1e815b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:32.420 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.415+0000 7fa1ecc75700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1e81598d0 0x7fa1e815b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52480/0 (socket says 172.21.15.27:52480) 2023-10-06T01:41:32.420 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.415+0000 7fa1ecc75700 1 -- 172.21.15.27:0/898106122 learned_addr learned my addr 172.21.15.27:0/898106122 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:32.420 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.415+0000 7fa1ed476700 1 --2- 172.21.15.27:0/898106122 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa1e8159500 0x7fa1e814c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:32.420 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.415+0000 7fa1edc77700 1 --2- 172.21.15.27:0/898106122 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa1e815d8e0 0x7fa1e815f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:32.420 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.416+0000 7fa1ecc75700 1 -- 172.21.15.27:0/898106122 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa1e815d8e0 msgr2=0x7fa1e815f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:32.421 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.416+0000 7fa1ecc75700 1 --2- 172.21.15.27:0/898106122 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa1e815d8e0 0x7fa1e815f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:32.421 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.416+0000 7fa1ecc75700 1 -- 172.21.15.27:0/898106122 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa1e8159500 msgr2=0x7fa1e814c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:32.421 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.416+0000 7fa1ecc75700 1 --2- 172.21.15.27:0/898106122 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa1e8159500 0x7fa1e814c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:32.421 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.416+0000 7fa1ecc75700 1 -- 172.21.15.27:0/898106122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1dc0094f0 con 0x7fa1e81598d0 2023-10-06T01:41:32.421 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.416+0000 7fa1ecc75700 1 --2- 172.21.15.27:0/898106122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1e81598d0 0x7fa1e815b0f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00ec50 tx=0x7fa1dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:32.421 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.416+0000 7fa1e57fa700 1 -- 172.21.15.27:0/898106122 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc0097d0 con 0x7fa1e81598d0 2023-10-06T01:41:32.421 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.416+0000 7fa1e57fa700 1 -- 172.21.15.27:0/898106122 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1dc00c9d0 con 0x7fa1e81598d0 2023-10-06T01:41:32.421 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.416+0000 7fa1ef6da700 1 -- 172.21.15.27:0/898106122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1d8009a20 con 0x7fa1e81598d0 2023-10-06T01:41:32.421 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.416+0000 7fa1e57fa700 1 -- 172.21.15.27:0/898106122 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc00cd50 con 0x7fa1e81598d0 2023-10-06T01:41:32.421 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.416+0000 7fa1ef6da700 1 -- 172.21.15.27:0/898106122 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1e814f280 con 0x7fa1e81598d0 2023-10-06T01:41:32.421 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.417+0000 7fa1e57fa700 1 -- 172.21.15.27:0/898106122 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa1dc013680 con 0x7fa1e81598d0 2023-10-06T01:41:32.421 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.418+0000 7fa1e57fa700 1 --2- 172.21.15.27:0/898106122 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa1d4066f30 0x7fa1d40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:32.422 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.418+0000 7fa1e57fa700 1 -- 172.21.15.27:0/898106122 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fa1dc084de0 con 0x7fa1e81598d0 2023-10-06T01:41:32.422 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.418+0000 7fa1ed476700 1 --2- 172.21.15.27:0/898106122 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa1d4066f30 0x7fa1d40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:32.422 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.418+0000 7fa1ed476700 1 --2- 172.21.15.27:0/898106122 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa1d4066f30 0x7fa1d40693e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa1e00011e0 tx=0x7fa1e0007bf0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:32.427 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.419+0000 7fa1f89d2b80 1 Processor -- start 2023-10-06T01:41:32.427 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.420+0000 7fa1f89d2b80 1 -- start start 2023-10-06T01:41:32.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.420+0000 7fa1f89d2b80 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x563be65d1840 0x563be65d1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:32.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.420+0000 7fa1f89d2b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563be65d2cc0 0x563be66216c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:32.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.420+0000 7fa1f89d2b80 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x563be6621c00 0x563be6622050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:32.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.420+0000 7fa1f89d2b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x563be6528a70 con 0x563be65d2cc0 2023-10-06T01:41:32.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.420+0000 7fa1f89d2b80 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x563be6526270 con 0x563be6621c00 2023-10-06T01:41:32.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.420+0000 7fa1f89d2b80 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x563be6522060 con 0x563be65d1840 2023-10-06T01:41:32.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.420+0000 7fa1ecc75700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563be65d2cc0 0x563be66216c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:32.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.420+0000 7fa1ecc75700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563be65d2cc0 0x563be66216c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52490/0 (socket says 172.21.15.27:52490) 2023-10-06T01:41:32.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.420+0000 7fa1ecc75700 1 -- 172.21.15.27:0/2333975293 learned_addr learned my addr 172.21.15.27:0/2333975293 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:32.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.420+0000 7fa1ed476700 1 --2- 172.21.15.27:0/2333975293 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x563be65d1840 0x563be65d1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:32.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.420+0000 7fa1edc77700 1 --2- 172.21.15.27:0/2333975293 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x563be6621c00 0x563be6622050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:32.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.421+0000 7fa1ecc75700 1 -- 172.21.15.27:0/2333975293 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x563be65d1840 msgr2=0x563be65d1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:32.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.421+0000 7fa1ecc75700 1 --2- 172.21.15.27:0/2333975293 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x563be65d1840 0x563be65d1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:32.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.421+0000 7fa1ecc75700 1 -- 172.21.15.27:0/2333975293 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x563be6621c00 msgr2=0x563be6622050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:32.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.421+0000 7fa1ecc75700 1 --2- 172.21.15.27:0/2333975293 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x563be6621c00 0x563be6622050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:32.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.421+0000 7fa1ecc75700 1 -- 172.21.15.27:0/2333975293 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563be63bc270 con 0x563be65d2cc0 2023-10-06T01:41:32.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.421+0000 7fa1edc77700 1 --2- 172.21.15.27:0/2333975293 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x563be6621c00 0x563be6622050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:32.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.421+0000 7fa1ecc75700 1 --2- 172.21.15.27:0/2333975293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563be65d2cc0 0x563be66216c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc083250 tx=0x7fa1dc083100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45868958f12d8ca4 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:32.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.421+0000 7fa1ce7fc700 1 -- 172.21.15.27:0/2333975293 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc058e00 con 0x563be65d2cc0 2023-10-06T01:41:32.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.421+0000 7fa1ce7fc700 1 -- 172.21.15.27:0/2333975293 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1dc05cb50 con 0x563be65d2cc0 2023-10-06T01:41:32.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.421+0000 7fa1ce7fc700 1 -- 172.21.15.27:0/2333975293 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc053f50 con 0x563be65d2cc0 2023-10-06T01:41:32.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.421+0000 7fa1f89d2b80 1 -- 172.21.15.27:0/2333975293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563be65d2cc0 msgr2=0x563be66216c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:32.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.421+0000 7fa1f89d2b80 1 --2- 172.21.15.27:0/2333975293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563be65d2cc0 0x563be66216c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc083250 tx=0x7fa1dc083100 comp rx=0 tx=0).stop 2023-10-06T01:41:32.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.421+0000 7fa1f89d2b80 1 -- 172.21.15.27:0/2333975293 shutdown_connections 2023-10-06T01:41:32.430 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.421+0000 7fa1f89d2b80 1 --2- 172.21.15.27:0/2333975293 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x563be65d1840 0x563be65d1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:32.430 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.421+0000 7fa1f89d2b80 1 --2- 172.21.15.27:0/2333975293 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x563be6621c00 0x563be6622050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:32.430 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.421+0000 7fa1f89d2b80 1 --2- 172.21.15.27:0/2333975293 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563be65d2cc0 0x563be66216c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:32.430 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.421+0000 7fa1f89d2b80 1 -- 172.21.15.27:0/2333975293 >> 172.21.15.27:0/2333975293 conn(0x563be65d3160 msgr2=0x563be65d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:32.430 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.422+0000 7fa1f89d2b80 1 -- 172.21.15.27:0/2333975293 shutdown_connections 2023-10-06T01:41:32.430 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.422+0000 7fa1f89d2b80 1 -- 172.21.15.27:0/2333975293 wait complete. 2023-10-06T01:41:32.430 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.422+0000 7fa1f89d2b80 1 -- start start 2023-10-06T01:41:32.430 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.422+0000 7fa1f89d2b80 1 Processor -- start 2023-10-06T01:41:32.430 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.423+0000 7fa1f89d2b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563be65d1840 0x563be65fed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:32.430 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.423+0000 7fa1f89d2b80 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x563be65d2cc0 0x563be65cce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:32.430 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.423+0000 7fa1f89d2b80 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x563be6621c00 0x563be6610c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:32.430 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.423+0000 7fa1f89d2b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x563be64cd890 con 0x563be65d1840 2023-10-06T01:41:32.431 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.423+0000 7fa1f89d2b80 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x563be64e8ea0 con 0x563be65d2cc0 2023-10-06T01:41:32.431 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.423+0000 7fa1f89d2b80 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x563be63a4e30 con 0x563be6621c00 2023-10-06T01:41:32.431 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.423+0000 7fa1ed476700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563be65d1840 0x563be65fed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:32.431 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.423+0000 7fa1f89d2b80 10 client.0 _ll_register_callbacks cb 0x563be6594e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-06T01:41:32.431 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.423+0000 7fa1ed476700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563be65d1840 0x563be65fed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52494/0 (socket says 172.21.15.27:52494) 2023-10-06T01:41:32.431 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.423+0000 7fa1ed476700 1 -- 172.21.15.27:0/2614925359 learned_addr learned my addr 172.21.15.27:0/2614925359 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:32.431 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.423+0000 7fa1ecc75700 1 --2- 172.21.15.27:0/2614925359 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x563be65d2cc0 0x563be65cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:32.431 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.423+0000 7fa1edc77700 1 --2- 172.21.15.27:0/2614925359 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x563be6621c00 0x563be6610c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:32.431 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.423+0000 7fa1ed476700 1 -- 172.21.15.27:0/2614925359 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x563be6621c00 msgr2=0x563be6610c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:32.431 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.423+0000 7fa1ed476700 1 --2- 172.21.15.27:0/2614925359 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x563be6621c00 0x563be6610c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:32.431 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.423+0000 7fa1ed476700 1 -- 172.21.15.27:0/2614925359 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x563be65d2cc0 msgr2=0x563be65cce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:32.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.423+0000 7fa1ed476700 1 --2- 172.21.15.27:0/2614925359 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x563be65d2cc0 0x563be65cce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:32.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.423+0000 7fa1ed476700 1 -- 172.21.15.27:0/2614925359 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1e814f280 con 0x563be65d1840 2023-10-06T01:41:32.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.423+0000 7fa1edc77700 1 --2- 172.21.15.27:0/2614925359 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x563be6621c00 0x563be6610c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:32.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.423+0000 7fa1ecc75700 1 --2- 172.21.15.27:0/2614925359 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x563be65d2cc0 0x563be65cce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:32.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.424+0000 7fa1ed476700 1 --2- 172.21.15.27:0/2614925359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563be65d1840 0x563be65fed00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0014e20 tx=0x7fa1e0013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41510964d93c6e25 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:32.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.424+0000 7fa1c77fe700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2023-10-06T01:41:32.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.424+0000 7fa1c77fe700 1 -- 172.21.15.27:0/2614925359 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e0014450 con 0x563be65d1840 2023-10-06T01:41:32.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.424+0000 7fa1c77fe700 1 -- 172.21.15.27:0/2614925359 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fa1e0001d40 con 0x563be65d1840 2023-10-06T01:41:32.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.424+0000 7fa1f89d2b80 1 -- 172.21.15.27:0/2614925359 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563be65c6940 con 0x563be65d1840 2023-10-06T01:41:32.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.424+0000 7fa1c77fe700 1 -- 172.21.15.27:0/2614925359 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa1e0021600 con 0x563be65d1840 2023-10-06T01:41:32.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.424+0000 7fa1f89d2b80 10 client.4671 fetch_fsmap learned FSMap version 7 2023-10-06T01:41:32.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.424+0000 7fa1f89d2b80 1 -- 172.21.15.27:0/2614925359 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x563be635efe0 con 0x563be65d1840 2023-10-06T01:41:32.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.424+0000 7fa1c77fe700 1 -- 172.21.15.27:0/2614925359 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x563be65c6940 con 0x563be65d1840 2023-10-06T01:41:32.433 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.424+0000 7fa1c77fe700 1 -- 172.21.15.27:0/2614925359 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 7) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fa1e0020070 con 0x563be65d1840 2023-10-06T01:41:32.433 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.424+0000 7fa1f89d2b80 10 client.4671 fetch_fsmap finished waiting for FSMap version 7 2023-10-06T01:41:32.433 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.424+0000 7fa1f89d2b80 10 client.4671 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-06T01:41:32.433 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.424+0000 7fa1f89d2b80 20 client.4671 populate_metadata read hostname 'smithi027' 2023-10-06T01:41:32.433 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.424+0000 7fa1f89d2b80 1 --2- 172.21.15.27:0/2614925359 >> [v2:172.21.15.124:6834/3803994994,v1:172.21.15.124:6835/3803994994] conn(0x563be662b670 0x563be662ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:32.433 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.424+0000 7fa1f89d2b80 4 client.4671 mds_command: new command op to 4305 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-06T01:41:32.433 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.424+0000 7fa1f89d2b80 1 -- 172.21.15.27:0/2614925359 --> [v2:172.21.15.124:6834/3803994994,v1:172.21.15.124:6835/3803994994] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563be65a3870 con 0x563be662b670 2023-10-06T01:41:32.433 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.425+0000 7fa1ecc75700 1 --2- 172.21.15.27:0/2614925359 >> [v2:172.21.15.124:6834/3803994994,v1:172.21.15.124:6835/3803994994] conn(0x563be662b670 0x563be662ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:32.433 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.425+0000 7fa1c77fe700 10 client.4671 ms_handle_connect on v2:172.21.15.124:6834/3803994994 2023-10-06T01:41:32.433 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.425+0000 7fa1ecc75700 1 --2- 172.21.15.27:0/2614925359 >> [v2:172.21.15.124:6834/3803994994,v1:172.21.15.124:6835/3803994994] conn(0x563be662b670 0x563be662ba40 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55f7475962fecb26 server_cookie=30e055a6ecadecb5 in_seq=0 out_seq=0 2023-10-06T01:41:32.433 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.425+0000 7fa1c77fe700 1 -- 172.21.15.27:0/2614925359 <== mds.0 v2:172.21.15.124:6834/3803994994 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x563be65a3870 con 0x563be662b670 2023-10-06T01:41:32.433 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.425+0000 7fa1c77fe700 10 client.4671 handle_command_reply: tid=0 2023-10-06T01:41:32.434 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.425+0000 7fa1f89d2b80 1 client.4671 shutdown 2023-10-06T01:41:32.434 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.426+0000 7fa1f89d2b80 1 -- 172.21.15.27:0/2614925359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563be65d1840 msgr2=0x563be65fed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:32.434 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.426+0000 7fa1f89d2b80 1 --2- 172.21.15.27:0/2614925359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563be65d1840 0x563be65fed00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0014e20 tx=0x7fa1e0013000 comp rx=0 tx=0).stop 2023-10-06T01:41:32.434 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.426+0000 7fa1f89d2b80 1 -- 172.21.15.27:0/2614925359 shutdown_connections 2023-10-06T01:41:32.434 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.426+0000 7fa1f89d2b80 1 --2- 172.21.15.27:0/2614925359 >> [v2:172.21.15.124:6834/3803994994,v1:172.21.15.124:6835/3803994994] conn(0x563be662b670 0x563be662ba40 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:32.434 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.426+0000 7fa1f89d2b80 1 --2- 172.21.15.27:0/2614925359 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x563be6621c00 0x563be6610c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:32.434 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.426+0000 7fa1f89d2b80 1 --2- 172.21.15.27:0/2614925359 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563be65d1840 0x563be65fed00 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:32.434 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.426+0000 7fa1c77fe700 0 client.4671 ms_handle_reset on v2:172.21.15.124:6834/3803994994 2023-10-06T01:41:32.434 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.426+0000 7fa1f89d2b80 1 --2- 172.21.15.27:0/2614925359 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x563be65d2cc0 0x563be65cce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:32.434 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.426+0000 7fa1f89d2b80 1 -- 172.21.15.27:0/2614925359 >> 172.21.15.27:0/2614925359 conn(0x563be65d3160 msgr2=0x563be65d21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:32.434 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.426+0000 7fa1f89d2b80 1 -- 172.21.15.27:0/2614925359 shutdown_connections 2023-10-06T01:41:32.434 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.426+0000 7fa1f89d2b80 1 -- 172.21.15.27:0/2614925359 wait complete. 2023-10-06T01:41:32.435 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.426+0000 7fa1f89d2b80 20 client.4671 trim_cache size 0 max 16384 2023-10-06T01:41:32.444 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.443+0000 7fa1f89d2b80 1 Processor -- start 2023-10-06T01:41:32.444 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.443+0000 7fa1f89d2b80 1 -- start start 2023-10-06T01:41:32.447 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.444+0000 7fa1f89d2b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563be65d1840 0x563be661b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:32.447 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.444+0000 7fa1f89d2b80 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x563be65d2cc0 0x563be65faa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:32.447 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.444+0000 7fa1f89d2b80 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x563be6621c00 0x563be65ff9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:32.447 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.444+0000 7fa1f89d2b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x563be6616fb0 con 0x563be65d1840 2023-10-06T01:41:32.447 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.444+0000 7fa1f89d2b80 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x563be66218e0 con 0x563be6621c00 2023-10-06T01:41:32.447 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.444+0000 7fa1f89d2b80 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x563be65d1c10 con 0x563be65d2cc0 2023-10-06T01:41:32.447 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.444+0000 7fa1ed476700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563be65d1840 0x563be661b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:32.448 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.444+0000 7fa1ed476700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563be65d1840 0x563be661b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52498/0 (socket says 172.21.15.27:52498) 2023-10-06T01:41:32.448 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.444+0000 7fa1ed476700 1 -- 172.21.15.27:0/2210145143 learned_addr learned my addr 172.21.15.27:0/2210145143 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:32.448 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.444+0000 7fa1edc77700 1 --2- 172.21.15.27:0/2210145143 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x563be6621c00 0x563be65ff9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:32.448 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.444+0000 7fa1ecc75700 1 --2- 172.21.15.27:0/2210145143 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x563be65d2cc0 0x563be65faa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:32.448 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.444+0000 7fa1ed476700 1 -- 172.21.15.27:0/2210145143 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x563be65d2cc0 msgr2=0x563be65faa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:32.448 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.444+0000 7fa1ed476700 1 --2- 172.21.15.27:0/2210145143 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x563be65d2cc0 0x563be65faa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:32.448 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.444+0000 7fa1ed476700 1 -- 172.21.15.27:0/2210145143 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x563be6621c00 msgr2=0x563be65ff9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:32.448 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.444+0000 7fa1ed476700 1 --2- 172.21.15.27:0/2210145143 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x563be6621c00 0x563be65ff9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:32.448 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.444+0000 7fa1ed476700 1 -- 172.21.15.27:0/2210145143 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563be64f0780 con 0x563be65d1840 2023-10-06T01:41:32.448 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.444+0000 7fa1ecc75700 1 --2- 172.21.15.27:0/2210145143 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x563be65d2cc0 0x563be65faa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:32.448 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.444+0000 7fa1edc77700 1 --2- 172.21.15.27:0/2210145143 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x563be6621c00 0x563be65ff9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:32.448 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.444+0000 7fa1ed476700 1 --2- 172.21.15.27:0/2210145143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563be65d1840 0x563be661b3d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0007ed0 tx=0x7fa1e0017030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=604da92b9406dac server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:32.449 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.444+0000 7fa1ceffd700 1 -- 172.21.15.27:0/2210145143 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e0021600 con 0x563be65d1840 2023-10-06T01:41:32.449 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.444+0000 7fa1ceffd700 1 -- 172.21.15.27:0/2210145143 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e0014030 con 0x563be65d1840 2023-10-06T01:41:32.449 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.444+0000 7fa1ceffd700 1 -- 172.21.15.27:0/2210145143 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e00143f0 con 0x563be65d1840 2023-10-06T01:41:32.449 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.445+0000 7fa1f89d2b80 1 -- 172.21.15.27:0/2210145143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563be65d1840 msgr2=0x563be661b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:32.450 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.445+0000 7fa1f89d2b80 1 --2- 172.21.15.27:0/2210145143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563be65d1840 0x563be661b3d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0007ed0 tx=0x7fa1e0017030 comp rx=0 tx=0).stop 2023-10-06T01:41:32.450 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.445+0000 7fa1f89d2b80 1 -- 172.21.15.27:0/2210145143 shutdown_connections 2023-10-06T01:41:32.450 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.445+0000 7fa1f89d2b80 1 --2- 172.21.15.27:0/2210145143 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x563be6621c00 0x563be65ff9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:32.450 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.445+0000 7fa1f89d2b80 1 --2- 172.21.15.27:0/2210145143 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563be65d1840 0x563be661b3d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:32.450 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.445+0000 7fa1f89d2b80 1 --2- 172.21.15.27:0/2210145143 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x563be65d2cc0 0x563be65faa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:32.450 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.445+0000 7fa1f89d2b80 1 -- 172.21.15.27:0/2210145143 >> 172.21.15.27:0/2210145143 conn(0x563be65d3160 msgr2=0x563be66151e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:32.450 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.445+0000 7fa1f89d2b80 1 -- 172.21.15.27:0/2210145143 shutdown_connections 2023-10-06T01:41:32.451 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.445+0000 7fa1f89d2b80 1 -- 172.21.15.27:0/2210145143 wait complete. 2023-10-06T01:41:32.451 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.446+0000 7fa1f89d2b80 1 -- start start 2023-10-06T01:41:32.451 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.446+0000 7fa1f89d2b80 1 Processor -- start 2023-10-06T01:41:32.454 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.447+0000 7fa1f89d2b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563be65d1840 0x563be65ce230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:32.454 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.447+0000 7fa1f89d2b80 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x563be65d2cc0 0x563be667d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:32.455 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.447+0000 7fa1f89d2b80 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x563be6621c00 0x563be6631a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:32.455 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.447+0000 7fa1f89d2b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x563be65cd550 con 0x563be65d1840 2023-10-06T01:41:32.455 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.447+0000 7fa1f89d2b80 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x563be65d4010 con 0x563be6621c00 2023-10-06T01:41:32.455 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.447+0000 7fa1f89d2b80 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x563be661fee0 con 0x563be65d2cc0 2023-10-06T01:41:32.455 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.447+0000 7fa1ed476700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563be65d1840 0x563be65ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:32.455 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.447+0000 7fa1f89d2b80 10 client.0 _ll_register_callbacks cb 0x563be6594e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-06T01:41:32.455 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.447+0000 7fa1ed476700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563be65d1840 0x563be65ce230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52506/0 (socket says 172.21.15.27:52506) 2023-10-06T01:41:32.455 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.447+0000 7fa1ed476700 1 -- 172.21.15.27:0/3452051664 learned_addr learned my addr 172.21.15.27:0/3452051664 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:32.455 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.447+0000 7fa1ecc75700 1 --2- 172.21.15.27:0/3452051664 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x563be65d2cc0 0x563be667d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:32.455 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.447+0000 7fa1edc77700 1 --2- 172.21.15.27:0/3452051664 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x563be6621c00 0x563be6631a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:32.455 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.447+0000 7fa1ed476700 1 -- 172.21.15.27:0/3452051664 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x563be65d2cc0 msgr2=0x563be667d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:32.455 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.447+0000 7fa1ed476700 1 --2- 172.21.15.27:0/3452051664 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x563be65d2cc0 0x563be667d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:32.456 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.447+0000 7fa1ed476700 1 -- 172.21.15.27:0/3452051664 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x563be6621c00 msgr2=0x563be6631a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:32.456 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.447+0000 7fa1ed476700 1 --2- 172.21.15.27:0/3452051664 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x563be6621c00 0x563be6631a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:32.456 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.447+0000 7fa1ed476700 1 -- 172.21.15.27:0/3452051664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563be635efe0 con 0x563be65d1840 2023-10-06T01:41:32.456 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.447+0000 7fa1ed476700 1 --2- 172.21.15.27:0/3452051664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563be65d1840 0x563be65ce230 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0023040 tx=0x7fa1e0034040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caaf5325ce3d99bb server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:32.456 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.447+0000 7fa1c77fe700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2023-10-06T01:41:32.456 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.448+0000 7fa1c77fe700 1 -- 172.21.15.27:0/3452051664 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e0001c90 con 0x563be65d1840 2023-10-06T01:41:32.456 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.448+0000 7fa1c77fe700 1 -- 172.21.15.27:0/3452051664 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fa1e0021b00 con 0x563be65d1840 2023-10-06T01:41:32.456 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.448+0000 7fa1f89d2b80 1 -- 172.21.15.27:0/3452051664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa1dc05cb50 con 0x563be65d1840 2023-10-06T01:41:32.456 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.448+0000 7fa1c77fe700 1 -- 172.21.15.27:0/3452051664 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa1e003b430 con 0x563be65d1840 2023-10-06T01:41:32.456 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.448+0000 7fa1f89d2b80 10 client.4677 fetch_fsmap learned FSMap version 7 2023-10-06T01:41:32.456 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.448+0000 7fa1f89d2b80 1 -- 172.21.15.27:0/3452051664 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x563be64d7470 con 0x563be65d1840 2023-10-06T01:41:32.456 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.449+0000 7fa1c77fe700 1 -- 172.21.15.27:0/3452051664 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1dc05cb50 con 0x563be65d1840 2023-10-06T01:41:32.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.449+0000 7fa1c77fe700 1 -- 172.21.15.27:0/3452051664 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 7) v1 ==== 3100+0+0 (secure 0 0 0) 0x7fa1e0044020 con 0x563be65d1840 2023-10-06T01:41:32.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.449+0000 7fa1f89d2b80 10 client.4677 fetch_fsmap finished waiting for FSMap version 7 2023-10-06T01:41:32.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.449+0000 7fa1f89d2b80 10 client.4677 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-06T01:41:32.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.449+0000 7fa1f89d2b80 20 client.4677 populate_metadata read hostname 'smithi027' 2023-10-06T01:41:32.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.449+0000 7fa1f89d2b80 1 --2- 172.21.15.27:0/3452051664 >> [v2:172.21.15.124:6834/3803994994,v1:172.21.15.124:6835/3803994994] conn(0x563be661f100 0x563be661f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:32.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.449+0000 7fa1f89d2b80 4 client.4677 mds_command: new command op to 4305 tid=0[{"prefix": "status"}] 2023-10-06T01:41:32.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.449+0000 7fa1f89d2b80 1 -- 172.21.15.27:0/3452051664 --> [v2:172.21.15.124:6834/3803994994,v1:172.21.15.124:6835/3803994994] -- command(tid 0: {"prefix": "status"}) v1 -- 0x563be64c4e00 con 0x563be661f100 2023-10-06T01:41:32.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.449+0000 7fa1ecc75700 1 --2- 172.21.15.27:0/3452051664 >> [v2:172.21.15.124:6834/3803994994,v1:172.21.15.124:6835/3803994994] conn(0x563be661f100 0x563be661f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:32.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.450+0000 7fa1c77fe700 10 client.4677 ms_handle_connect on v2:172.21.15.124:6834/3803994994 2023-10-06T01:41:32.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.450+0000 7fa1ecc75700 1 --2- 172.21.15.27:0/3452051664 >> [v2:172.21.15.124:6834/3803994994,v1:172.21.15.124:6835/3803994994] conn(0x563be661f100 0x563be661f4d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ceb5c9b0c6e1fd5 server_cookie=8b60190df69bc28e in_seq=0 out_seq=0 2023-10-06T01:41:32.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.450+0000 7fa1c77fe700 1 -- 172.21.15.27:0/3452051664 <== mds.0 v2:172.21.15.124:6834/3803994994 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x563be64c4e00 con 0x563be661f100 2023-10-06T01:41:32.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.450+0000 7fa1c77fe700 10 client.4677 handle_command_reply: tid=0 2023-10-06T01:41:32.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.450+0000 7fa1f89d2b80 1 client.4677 shutdown 2023-10-06T01:41:32.458 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.450+0000 7fa1f89d2b80 1 -- 172.21.15.27:0/3452051664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563be65d1840 msgr2=0x563be65ce230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:32.458 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.450+0000 7fa1f89d2b80 1 --2- 172.21.15.27:0/3452051664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563be65d1840 0x563be65ce230 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0023040 tx=0x7fa1e0034040 comp rx=0 tx=0).stop 2023-10-06T01:41:32.458 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.450+0000 7fa1f89d2b80 1 -- 172.21.15.27:0/3452051664 shutdown_connections 2023-10-06T01:41:32.458 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.450+0000 7fa1f89d2b80 1 --2- 172.21.15.27:0/3452051664 >> [v2:172.21.15.124:6834/3803994994,v1:172.21.15.124:6835/3803994994] conn(0x563be661f100 0x563be661f4d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:32.458 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.450+0000 7fa1f89d2b80 1 --2- 172.21.15.27:0/3452051664 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x563be6621c00 0x563be6631a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:32.458 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.450+0000 7fa1c77fe700 0 client.4677 ms_handle_reset on v2:172.21.15.124:6834/3803994994 2023-10-06T01:41:32.458 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.450+0000 7fa1f89d2b80 1 --2- 172.21.15.27:0/3452051664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x563be65d1840 0x563be65ce230 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:32.458 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.450+0000 7fa1f89d2b80 1 --2- 172.21.15.27:0/3452051664 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x563be65d2cc0 0x563be667d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:32.458 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.450+0000 7fa1f89d2b80 1 -- 172.21.15.27:0/3452051664 >> 172.21.15.27:0/3452051664 conn(0x563be65d3160 msgr2=0x563be6610890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:32.458 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.450+0000 7fa1f89d2b80 1 -- 172.21.15.27:0/3452051664 shutdown_connections 2023-10-06T01:41:32.458 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.450+0000 7fa1f89d2b80 1 -- 172.21.15.27:0/3452051664 wait complete. 2023-10-06T01:41:32.459 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.450+0000 7fa1f89d2b80 20 client.4677 trim_cache size 0 max 16384 2023-10-06T01:41:32.459 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2023-10-06T01:41:32.459 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "20a7bff0-74f8-4f2f-b6c6-899e804bc965", 2023-10-06T01:41:32.459 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2023-10-06T01:41:32.459 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 4305, 2023-10-06T01:41:32.459 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2023-10-06T01:41:32.459 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2023-10-06T01:41:32.459 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2023-10-06T01:41:32.459 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 19.126517477, 2023-10-06T01:41:32.460 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 7, 2023-10-06T01:41:32.460 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 20, 2023-10-06T01:41:32.460 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 20, 2023-10-06T01:41:32.460 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 25.084545886000001 2023-10-06T01:41:32.460 INFO:teuthology.orchestra.run.smithi027.stdout:} 2023-10-06T01:41:32.461 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.451+0000 7fa1ef6da700 1 -- 172.21.15.27:0/898106122 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa1d4066f30 msgr2=0x7fa1d40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:32.461 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.451+0000 7fa1ef6da700 1 --2- 172.21.15.27:0/898106122 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa1d4066f30 0x7fa1d40693e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fa1e00011e0 tx=0x7fa1e0007bf0 comp rx=0 tx=0).stop 2023-10-06T01:41:32.461 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.451+0000 7fa1ef6da700 1 -- 172.21.15.27:0/898106122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1e81598d0 msgr2=0x7fa1e815b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:32.461 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.451+0000 7fa1ef6da700 1 --2- 172.21.15.27:0/898106122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1e81598d0 0x7fa1e815b0f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00ec50 tx=0x7fa1dc009250 comp rx=0 tx=0).stop 2023-10-06T01:41:32.461 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.451+0000 7fa1ef6da700 1 -- 172.21.15.27:0/898106122 shutdown_connections 2023-10-06T01:41:32.461 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.451+0000 7fa1ef6da700 1 --2- 172.21.15.27:0/898106122 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa1d4066f30 0x7fa1d40693e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:32.461 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.451+0000 7fa1ef6da700 1 --2- 172.21.15.27:0/898106122 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa1e815d8e0 0x7fa1e815f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:32.461 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.451+0000 7fa1ef6da700 1 --2- 172.21.15.27:0/898106122 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa1e8159500 0x7fa1e814c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:32.461 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.451+0000 7fa1ef6da700 1 --2- 172.21.15.27:0/898106122 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1e81598d0 0x7fa1e815b0f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:32.461 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.451+0000 7fa1ef6da700 1 -- 172.21.15.27:0/898106122 >> 172.21.15.27:0/898106122 conn(0x7fa1e8067630 msgr2=0x7fa1e80ceb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:32.462 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.451+0000 7fa1ef6da700 1 -- 172.21.15.27:0/898106122 shutdown_connections 2023-10-06T01:41:32.462 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:32.451+0000 7fa1ef6da700 1 -- 172.21.15.27:0/898106122 wait complete. 2023-10-06T01:41:32.464 INFO:teuthology.run_tasks:Running task kclient... 2023-10-06T01:41:32.486 INFO:tasks.kclient:Mounting kernel clients... 2023-10-06T01:41:32.486 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2023-10-06T01:41:32.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:32.486 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2023-10-06T01:41:32.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:32.501 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link delete ceph-brx 2023-10-06T01:41:32.568 INFO:teuthology.orchestra.run.smithi027.stderr:Cannot find device "ceph-brx" 2023-10-06T01:41:32.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T01:41:32.569 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:32.570 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2023-10-06T01:41:32.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:32.625 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link delete ceph-brx 2023-10-06T01:41:32.690 INFO:teuthology.orchestra.run.smithi027.stderr:Cannot find device "ceph-brx" 2023-10-06T01:41:32.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T01:41:32.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:32.692 DEBUG:teuthology.orchestra.run.smithi124:> ip netns list 2023-10-06T01:41:32.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:32.710 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link delete ceph-brx 2023-10-06T01:41:32.780 INFO:teuthology.orchestra.run.smithi124.stderr:Cannot find device "ceph-brx" 2023-10-06T01:41:32.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T01:41:32.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:32.782 DEBUG:teuthology.orchestra.run.smithi124:> ip netns list 2023-10-06T01:41:32.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:32.839 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link delete ceph-brx 2023-10-06T01:41:32.907 INFO:teuthology.orchestra.run.smithi124.stderr:Cannot find device "ceph-brx" 2023-10-06T01:41:32.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T01:41:32.908 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2023-10-06T01:41:32.909 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-06T01:41:33.027 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.026+0000 7fdd6885a700 1 Processor -- start 2023-10-06T01:41:33.028 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.026+0000 7fdd6885a700 1 -- start start 2023-10-06T01:41:33.030 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.027+0000 7fdd6885a700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdd6415d560 0x7fdd640d0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:33.030 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.027+0000 7fdd6885a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdd6415d930 0x7fdd640b1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:33.030 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.027+0000 7fdd6885a700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fdd641591b0 0x7fdd64159630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:33.030 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.027+0000 7fdd6885a700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd640b2ad0 con 0x7fdd6415d930 2023-10-06T01:41:33.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.027+0000 7fdd6885a700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd640b4160 con 0x7fdd641591b0 2023-10-06T01:41:33.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.027+0000 7fdd6885a700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd640b1f80 con 0x7fdd6415d560 2023-10-06T01:41:33.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.027+0000 7fdd61d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdd6415d930 0x7fdd640b1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:33.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.027+0000 7fdd61d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdd6415d930 0x7fdd640b1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52510/0 (socket says 172.21.15.27:52510) 2023-10-06T01:41:33.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.027+0000 7fdd61d9b700 1 -- 172.21.15.27:0/944793290 learned_addr learned my addr 172.21.15.27:0/944793290 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:33.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.028+0000 7fdd62d9d700 1 --2- 172.21.15.27:0/944793290 >> v2:172.21.15.124:3300/0 conn(0x7fdd641591b0 0x7fdd64159630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:33.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.028+0000 7fdd6259c700 1 --2- 172.21.15.27:0/944793290 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdd6415d560 0x7fdd640d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:33.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.028+0000 7fdd61d9b700 1 -- 172.21.15.27:0/944793290 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdd6415d560 msgr2=0x7fdd640d0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:33.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.028+0000 7fdd61d9b700 1 --2- 172.21.15.27:0/944793290 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdd6415d560 0x7fdd640d0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:33.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.028+0000 7fdd61d9b700 1 -- 172.21.15.27:0/944793290 >> v2:172.21.15.124:3300/0 conn(0x7fdd641591b0 msgr2=0x7fdd64159630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:33.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.028+0000 7fdd61d9b700 1 --2- 172.21.15.27:0/944793290 >> v2:172.21.15.124:3300/0 conn(0x7fdd641591b0 0x7fdd64159630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:33.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.028+0000 7fdd61d9b700 1 -- 172.21.15.27:0/944793290 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd64159b70 con 0x7fdd6415d930 2023-10-06T01:41:33.032 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.028+0000 7fdd61d9b700 1 --2- 172.21.15.27:0/944793290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd6415d930 0x7fdd640b1a40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c0099b0 tx=0x7fdd4c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f051ecede3f00f71 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:33.032 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.028+0000 7fdd6159a700 1 -- 172.21.15.27:0/944793290 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c00b820 con 0x7fdd6415d930 2023-10-06T01:41:33.032 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.028+0000 7fdd6159a700 1 -- 172.21.15.27:0/944793290 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd4c00b980 con 0x7fdd6415d930 2023-10-06T01:41:33.032 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.028+0000 7fdd6159a700 1 -- 172.21.15.27:0/944793290 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c00bd20 con 0x7fdd6415d930 2023-10-06T01:41:33.032 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.028+0000 7fdd6885a700 1 -- 172.21.15.27:0/944793290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd6415d930 msgr2=0x7fdd640b1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:33.032 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.028+0000 7fdd6885a700 1 --2- 172.21.15.27:0/944793290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd6415d930 0x7fdd640b1a40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c0099b0 tx=0x7fdd4c009250 comp rx=0 tx=0).stop 2023-10-06T01:41:33.032 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.029+0000 7fdd6885a700 1 -- 172.21.15.27:0/944793290 shutdown_connections 2023-10-06T01:41:33.032 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.029+0000 7fdd6885a700 1 --2- 172.21.15.27:0/944793290 >> v2:172.21.15.124:3300/0 conn(0x7fdd641591b0 0x7fdd64159630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:33.032 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.029+0000 7fdd6885a700 1 --2- 172.21.15.27:0/944793290 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdd6415d560 0x7fdd640d0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:33.032 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.029+0000 7fdd6885a700 1 --2- 172.21.15.27:0/944793290 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd6415d930 0x7fdd640b1a40 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:33.032 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.029+0000 7fdd6885a700 1 -- 172.21.15.27:0/944793290 >> 172.21.15.27:0/944793290 conn(0x7fdd640d1180 msgr2=0x7fdd64069640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:33.033 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.029+0000 7fdd6885a700 1 -- 172.21.15.27:0/944793290 shutdown_connections 2023-10-06T01:41:33.033 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.029+0000 7fdd6885a700 1 -- 172.21.15.27:0/944793290 wait complete. 2023-10-06T01:41:33.035 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.030+0000 7fdd6885a700 1 Processor -- start 2023-10-06T01:41:33.035 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.030+0000 7fdd6885a700 1 -- start start 2023-10-06T01:41:33.035 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.030+0000 7fdd6885a700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdd641591b0 0x7fdd64150d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:33.035 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.030+0000 7fdd6885a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd6415d560 0x7fdd640b5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:33.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.030+0000 7fdd6885a700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdd6415d930 0x7fdd640b5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:33.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.030+0000 7fdd6885a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6415a040 con 0x7fdd6415d560 2023-10-06T01:41:33.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.030+0000 7fdd6885a700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6414fb70 con 0x7fdd641591b0 2023-10-06T01:41:33.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.030+0000 7fdd6885a700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd64151400 con 0x7fdd6415d930 2023-10-06T01:41:33.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.031+0000 7fdd61d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd6415d560 0x7fdd640b5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:33.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.031+0000 7fdd61d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd6415d560 0x7fdd640b5a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52522/0 (socket says 172.21.15.27:52522) 2023-10-06T01:41:33.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.031+0000 7fdd61d9b700 1 -- 172.21.15.27:0/2118486014 learned_addr learned my addr 172.21.15.27:0/2118486014 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:33.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.031+0000 7fdd6259c700 1 --2- 172.21.15.27:0/2118486014 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdd641591b0 0x7fdd64150d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:33.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.031+0000 7fdd62d9d700 1 --2- 172.21.15.27:0/2118486014 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdd6415d930 0x7fdd640b5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:33.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.031+0000 7fdd61d9b700 1 -- 172.21.15.27:0/2118486014 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdd6415d930 msgr2=0x7fdd640b5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:33.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.031+0000 7fdd61d9b700 1 --2- 172.21.15.27:0/2118486014 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdd6415d930 0x7fdd640b5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:33.037 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.031+0000 7fdd61d9b700 1 -- 172.21.15.27:0/2118486014 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdd641591b0 msgr2=0x7fdd64150d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:33.037 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.031+0000 7fdd61d9b700 1 --2- 172.21.15.27:0/2118486014 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdd641591b0 0x7fdd64150d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:33.037 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.031+0000 7fdd61d9b700 1 -- 172.21.15.27:0/2118486014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd4c0094f0 con 0x7fdd6415d560 2023-10-06T01:41:33.037 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.031+0000 7fdd61d9b700 1 --2- 172.21.15.27:0/2118486014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd6415d560 0x7fdd640b5a10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c009fd0 tx=0x7fdd4c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:33.037 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.031+0000 7fdd5a7fc700 1 -- 172.21.15.27:0/2118486014 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c011d70 con 0x7fdd6415d560 2023-10-06T01:41:33.037 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.031+0000 7fdd5a7fc700 1 -- 172.21.15.27:0/2118486014 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd4c024440 con 0x7fdd6415d560 2023-10-06T01:41:33.037 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.031+0000 7fdd6885a700 1 -- 172.21.15.27:0/2118486014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd641516e0 con 0x7fdd6415d560 2023-10-06T01:41:33.037 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.031+0000 7fdd5a7fc700 1 -- 172.21.15.27:0/2118486014 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c024800 con 0x7fdd6415d560 2023-10-06T01:41:33.037 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.031+0000 7fdd6885a700 1 -- 172.21.15.27:0/2118486014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd64151ca0 con 0x7fdd6415d560 2023-10-06T01:41:33.037 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.032+0000 7fdd6885a700 1 -- 172.21.15.27:0/2118486014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd6415a1f0 con 0x7fdd6415d560 2023-10-06T01:41:33.037 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.033+0000 7fdd5a7fc700 1 -- 172.21.15.27:0/2118486014 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fdd4c02d440 con 0x7fdd6415d560 2023-10-06T01:41:33.037 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.033+0000 7fdd5a7fc700 1 --2- 172.21.15.27:0/2118486014 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdd44066f30 0x7fdd440693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:33.037 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.033+0000 7fdd5a7fc700 1 -- 172.21.15.27:0/2118486014 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fdd4c086000 con 0x7fdd6415d560 2023-10-06T01:41:33.038 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.033+0000 7fdd6259c700 1 --2- 172.21.15.27:0/2118486014 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdd44066f30 0x7fdd440693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:33.038 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.034+0000 7fdd6259c700 1 --2- 172.21.15.27:0/2118486014 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdd44066f30 0x7fdd440693e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c006fd0 tx=0x7fdd5c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:33.038 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.037+0000 7fdd5a7fc700 1 -- 172.21.15.27:0/2118486014 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdd6415a1f0 con 0x7fdd6415d560 2023-10-06T01:41:33.275 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.273+0000 7fdd6885a700 1 -- 172.21.15.27:0/2118486014 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fdd64064050 con 0x7fdd6415d560 2023-10-06T01:41:33.275 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.274+0000 7fdd5a7fc700 1 -- 172.21.15.27:0/2118486014 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7fdd4c053110 con 0x7fdd6415d560 2023-10-06T01:41:33.275 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-06T01:41:33.277 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.276+0000 7fdd6885a700 1 -- 172.21.15.27:0/2118486014 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdd44066f30 msgr2=0x7fdd440693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:33.277 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.276+0000 7fdd6885a700 1 --2- 172.21.15.27:0/2118486014 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdd44066f30 0x7fdd440693e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c006fd0 tx=0x7fdd5c008040 comp rx=0 tx=0).stop 2023-10-06T01:41:33.278 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.276+0000 7fdd6885a700 1 -- 172.21.15.27:0/2118486014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd6415d560 msgr2=0x7fdd640b5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:33.278 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.276+0000 7fdd6885a700 1 --2- 172.21.15.27:0/2118486014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd6415d560 0x7fdd640b5a10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c009fd0 tx=0x7fdd4c017000 comp rx=0 tx=0).stop 2023-10-06T01:41:33.278 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.276+0000 7fdd6885a700 1 -- 172.21.15.27:0/2118486014 shutdown_connections 2023-10-06T01:41:33.278 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.276+0000 7fdd6885a700 1 --2- 172.21.15.27:0/2118486014 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdd44066f30 0x7fdd440693e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:33.278 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.276+0000 7fdd6885a700 1 --2- 172.21.15.27:0/2118486014 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdd6415d930 0x7fdd640b5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:33.278 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.276+0000 7fdd6885a700 1 --2- 172.21.15.27:0/2118486014 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdd641591b0 0x7fdd64150d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:33.278 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.276+0000 7fdd6885a700 1 --2- 172.21.15.27:0/2118486014 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdd6415d560 0x7fdd640b5a10 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:33.278 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.276+0000 7fdd6885a700 1 -- 172.21.15.27:0/2118486014 >> 172.21.15.27:0/2118486014 conn(0x7fdd640d1180 msgr2=0x7fdd640b1390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:33.278 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.276+0000 7fdd6885a700 1 -- 172.21.15.27:0/2118486014 shutdown_connections 2023-10-06T01:41:33.278 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:33.276+0000 7fdd6885a700 1 -- 172.21.15.27:0/2118486014 wait complete. 2023-10-06T01:41:33.289 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-06T01:41:33.289 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-06T01:41:33.289 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2023-10-06T01:41:33.289 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-06T01:41:33.289 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-06T01:41:33.289 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-06T01:41:33.289 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-06T01:41:33.289 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-06T01:41:33.289 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-06T01:41:33.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:33.290 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2023-10-06T01:41:33.305 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-06T01:41:33.305 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-06T01:41:33.305 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2023-10-06T01:41:33.305 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:41:33.305 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2023-10-06T01:41:33.305 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:41:33.305 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-06T01:41:33.306 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2023-10-06T01:41:33.306 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2023-10-06T01:41:33.306 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-06T01:41:33.306 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2023-10-06T01:41:33.306 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2023-10-06T01:41:33.306 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-06T01:41:33.306 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2023-10-06T01:41:33.306 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2023-10-06T01:41:33.306 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-06T01:41:33.306 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 42478sec preferred_lft 42478sec 2023-10-06T01:41:33.307 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link 2023-10-06T01:41:33.307 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:41:33.307 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-06T01:41:33.307 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2023-10-06T01:41:33.307 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2023-10-06T01:41:33.307 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-06T01:41:33.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:33.308 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2023-10-06T01:41:33.308 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2023-10-06T01:41:33.308 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link add name ceph-brx type bridge 2023-10-06T01:41:33.308 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip addr flush dev ceph-brx 2023-10-06T01:41:33.308 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set ceph-brx up 2023-10-06T01:41:33.308 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-06T01:41:33.308 DEBUG:teuthology.orchestra.run.smithi027:> ') 2023-10-06T01:41:33.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:33.489 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-06T01:41:33.535 INFO:teuthology.orchestra.run.smithi027.stdout:1 2023-10-06T01:41:33.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:33.536 DEBUG:teuthology.orchestra.run.smithi027:> route 2023-10-06T01:41:33.567 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2023-10-06T01:41:33.568 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-06T01:41:33.568 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-06T01:41:33.568 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-06T01:41:33.568 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-06T01:41:33.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:33.568 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2023-10-06T01:41:33.569 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2023-10-06T01:41:33.569 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-06T01:41:33.569 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-06T01:41:33.569 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-06T01:41:33.569 DEBUG:teuthology.orchestra.run.smithi027:> ') 2023-10-06T01:41:33.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:33.819 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2023-10-06T01:41:33.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:33.834 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list-id 2023-10-06T01:41:33.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:33.889 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2023-10-06T01:41:33.889 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2023-10-06T01:41:33.889 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-06T01:41:33.889 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-10-06T01:41:33.889 DEBUG:teuthology.orchestra.run.smithi027:> ') 2023-10-06T01:41:33.980 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-10-06T01:41:33.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:33.981 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2023-10-06T01:41:33.981 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2023-10-06T01:41:33.981 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-10-06T01:41:33.981 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-06T01:41:33.981 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-10-06T01:41:33.981 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-10-06T01:41:33.981 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-10-06T01:41:33.981 DEBUG:teuthology.orchestra.run.smithi027:> ') 2023-10-06T01:41:34.170 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:34.171 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2023-10-06T01:41:34.171 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2023-10-06T01:41:34.171 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2023-10-06T01:41:34.171 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set dev brx.0 master ceph-brx 2023-10-06T01:41:34.171 DEBUG:teuthology.orchestra.run.smithi027:> ') 2023-10-06T01:41:34.264 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T01:41:34.265 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-06T01:41:34.279 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-06T01:41:34.280 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T01:41:34.280 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-06T01:41:34.335 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-06T01:41:34.335 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-10-06T01:41:34.335 DEBUG:teuthology.orchestra.run.smithi027:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-06T01:41:34.428 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-06T01:41:34.428 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-06T01:41:34.428 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2023-10-06T01:41:34.511 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-06T01:41:34.511 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-06T01:41:34.511 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2023-10-06T01:41:34.511 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.124:3300,172.21.15.124:3301" 2023-10-06T01:41:34.511 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-06T01:41:34.535 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-06T01:41:34.535 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-06T01:41:34.535 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2023-10-06T01:41:34.535 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.124:3300,172.21.15.124:3301" 2023-10-06T01:41:34.535 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-06T01:41:34.535 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:6789,172.21.15.124:6789,172.21.15.124:6790" 2023-10-06T01:41:34.535 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.27:6789,172.21.15.124:6789,172.21.15.124:6790:/ 2023-10-06T01:41:34.535 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=20a7bff0-74f8-4f2f-b6c6-899e804bc965" will pass to kernel 2023-10-06T01:41:34.537 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-06T01:41:34.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:34.538 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-06T01:41:34.563 INFO:tasks.kclient:client.2 config is {'syntax': 'v1'} 2023-10-06T01:41:34.564 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-06T01:41:34.725 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.723+0000 7f71969ae700 1 Processor -- start 2023-10-06T01:41:34.725 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.723+0000 7f71969ae700 1 -- start start 2023-10-06T01:41:34.726 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.724+0000 7f71969ae700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f71900aca50 0x7f7190068a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:34.726 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.724+0000 7f71969ae700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71900b0920 con 0x7f7190068640 2023-10-06T01:41:34.727 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.724+0000 7f71969ae700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71900abeb0 con 0x7f71901501f0 2023-10-06T01:41:34.727 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.724+0000 7f71969ae700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7190068fd0 con 0x7f71900aca50 2023-10-06T01:41:34.727 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.724+0000 7f7194f4b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f71900aca50 0x7f7190068a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:34.727 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.724+0000 7f7187fff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7190068640 0x7f71900ac340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:57114/0 (socket says 172.21.15.27:57114) 2023-10-06T01:41:34.727 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.724+0000 7f7187fff700 1 -- 172.21.15.27:0/114287188 learned_addr learned my addr 172.21.15.27:0/114287188 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:34.727 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.724+0000 7f7194f4b700 1 --2- 172.21.15.27:0/114287188 >> v2:172.21.15.27:3300/0 conn(0x7f71900aca50 0x7f7190068a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52534/0 (socket says 172.21.15.27:52534) 2023-10-06T01:41:34.727 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.725+0000 7f7194f4b700 1 -- 172.21.15.27:0/114287188 >> v1:172.21.15.124:6789/0 conn(0x7f71901501f0 legacy=0x7f71900ad590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:34.727 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.725+0000 7f7194f4b700 1 -- 172.21.15.27:0/114287188 >> v1:172.21.15.27:6789/0 conn(0x7f7190068640 legacy=0x7f71900ac340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:34.727 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.725+0000 7f718f7fe700 1 -- 172.21.15.27:0/114287188 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1676924650 0 0) 0x7f71900b0920 con 0x7f7190068640 2023-10-06T01:41:34.727 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.725+0000 7f7194f4b700 1 -- 172.21.15.27:0/114287188 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71900b0640 con 0x7f71900aca50 2023-10-06T01:41:34.727 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.725+0000 7f7194f4b700 1 --2- 172.21.15.27:0/114287188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71900aca50 0x7f7190068a90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f718800af70 tx=0x7f718800a890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee0d71e93130eeab server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:34.728 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.725+0000 7f718f7fe700 1 -- 172.21.15.27:0/114287188 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f718800f070 con 0x7f71900aca50 2023-10-06T01:41:34.728 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.725+0000 7f718f7fe700 1 -- 172.21.15.27:0/114287188 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f718800a2e0 con 0x7f71900aca50 2023-10-06T01:41:34.728 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.725+0000 7f718f7fe700 1 -- 172.21.15.27:0/114287188 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f718800cd20 con 0x7f71900aca50 2023-10-06T01:41:34.730 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.726+0000 7f71969ae700 1 -- 172.21.15.27:0/114287188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71900aca50 msgr2=0x7f7190068a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:34.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.726+0000 7f71969ae700 1 --2- 172.21.15.27:0/114287188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71900aca50 0x7f7190068a90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f718800af70 tx=0x7f718800a890 comp rx=0 tx=0).stop 2023-10-06T01:41:34.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.726+0000 7f71969ae700 1 -- 172.21.15.27:0/114287188 shutdown_connections 2023-10-06T01:41:34.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.726+0000 7f71969ae700 1 --2- 172.21.15.27:0/114287188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71900aca50 0x7f7190068a90 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:34.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.726+0000 7f71969ae700 1 -- 172.21.15.27:0/114287188 >> 172.21.15.27:0/114287188 conn(0x7f71900ccd30 msgr2=0x7f71900694b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:34.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.726+0000 7f71969ae700 1 -- 172.21.15.27:0/114287188 shutdown_connections 2023-10-06T01:41:34.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.726+0000 7f71969ae700 1 -- 172.21.15.27:0/114287188 wait complete. 2023-10-06T01:41:34.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.727+0000 7f71969ae700 1 Processor -- start 2023-10-06T01:41:34.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.727+0000 7f71969ae700 1 -- start start 2023-10-06T01:41:34.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.728+0000 7f71969ae700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7190068640 0x7f7190144eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:34.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.728+0000 7f71969ae700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f71900aca50 0x7f719014ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:34.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.728+0000 7f71969ae700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71901501f0 0x7f719014d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:34.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.728+0000 7f71969ae700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719014ddb0 con 0x7f71901501f0 2023-10-06T01:41:34.732 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.728+0000 7f71969ae700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f719014c590 con 0x7f7190068640 2023-10-06T01:41:34.732 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.728+0000 7f71969ae700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71901453f0 con 0x7f71900aca50 2023-10-06T01:41:34.732 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.728+0000 7f7194f4b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71901501f0 0x7f719014d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:34.732 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.728+0000 7f7194f4b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71901501f0 0x7f719014d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:52540/0 (socket says 172.21.15.27:52540) 2023-10-06T01:41:34.732 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.728+0000 7f7194f4b700 1 -- 172.21.15.27:0/2148114346 learned_addr learned my addr 172.21.15.27:0/2148114346 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:34.732 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.728+0000 7f7187fff700 1 --2- 172.21.15.27:0/2148114346 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f71900aca50 0x7f719014ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:34.732 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.728+0000 7f718ffff700 1 --2- 172.21.15.27:0/2148114346 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7190068640 0x7f7190144eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:34.732 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.728+0000 7f7194f4b700 1 -- 172.21.15.27:0/2148114346 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f71900aca50 msgr2=0x7f719014ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:34.732 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.728+0000 7f7194f4b700 1 --2- 172.21.15.27:0/2148114346 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f71900aca50 0x7f719014ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:34.732 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.728+0000 7f7194f4b700 1 -- 172.21.15.27:0/2148114346 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7190068640 msgr2=0x7f7190144eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:34.732 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.728+0000 7f7194f4b700 1 --2- 172.21.15.27:0/2148114346 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7190068640 0x7f7190144eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:34.733 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.728+0000 7f7194f4b700 1 -- 172.21.15.27:0/2148114346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f718800ab30 con 0x7f71901501f0 2023-10-06T01:41:34.733 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.728+0000 7f718ffff700 1 --2- 172.21.15.27:0/2148114346 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7190068640 0x7f7190144eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:34.733 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.728+0000 7f7194f4b700 1 --2- 172.21.15.27:0/2148114346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71901501f0 0x7f719014d310 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f71880099a0 tx=0x7f7188019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:34.733 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.728+0000 7f718cff9700 1 -- 172.21.15.27:0/2148114346 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7188009e40 con 0x7f71901501f0 2023-10-06T01:41:34.733 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.728+0000 7f718cff9700 1 -- 172.21.15.27:0/2148114346 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7188027410 con 0x7f71901501f0 2023-10-06T01:41:34.733 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.728+0000 7f71969ae700 1 -- 172.21.15.27:0/2148114346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71900cbc80 con 0x7f71901501f0 2023-10-06T01:41:34.733 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.728+0000 7f718cff9700 1 -- 172.21.15.27:0/2148114346 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71880277d0 con 0x7f71901501f0 2023-10-06T01:41:34.733 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.728+0000 7f71969ae700 1 -- 172.21.15.27:0/2148114346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f719014bf40 con 0x7f71901501f0 2023-10-06T01:41:34.733 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.729+0000 7f71969ae700 1 -- 172.21.15.27:0/2148114346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7190064050 con 0x7f71901501f0 2023-10-06T01:41:34.734 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.730+0000 7f718cff9700 1 -- 172.21.15.27:0/2148114346 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f718801d410 con 0x7f71901501f0 2023-10-06T01:41:34.734 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.730+0000 7f718cff9700 1 --2- 172.21.15.27:0/2148114346 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7170066f30 0x7f71700693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:34.734 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.730+0000 7f718cff9700 1 -- 172.21.15.27:0/2148114346 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f7188086630 con 0x7f71901501f0 2023-10-06T01:41:34.734 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.731+0000 7f718ffff700 1 --2- 172.21.15.27:0/2148114346 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7170066f30 0x7f71700693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:34.734 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.731+0000 7f718ffff700 1 --2- 172.21.15.27:0/2148114346 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7170066f30 0x7f71700693e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7180007fd0 tx=0x7f7180009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:34.736 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.735+0000 7f718cff9700 1 -- 172.21.15.27:0/2148114346 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7188053740 con 0x7f71901501f0 2023-10-06T01:41:34.975 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.974+0000 7f71969ae700 1 -- 172.21.15.27:0/2148114346 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f719014c6d0 con 0x7f71901501f0 2023-10-06T01:41:34.975 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.974+0000 7f718cff9700 1 -- 172.21.15.27:0/2148114346 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7f7188056d60 con 0x7f71901501f0 2023-10-06T01:41:34.976 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-06T01:41:34.977 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.976+0000 7f71969ae700 1 -- 172.21.15.27:0/2148114346 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7170066f30 msgr2=0x7f71700693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:34.977 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.976+0000 7f71969ae700 1 --2- 172.21.15.27:0/2148114346 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7170066f30 0x7f71700693e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7180007fd0 tx=0x7f7180009040 comp rx=0 tx=0).stop 2023-10-06T01:41:34.978 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.976+0000 7f71969ae700 1 -- 172.21.15.27:0/2148114346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71901501f0 msgr2=0x7f719014d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:34.978 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.976+0000 7f71969ae700 1 --2- 172.21.15.27:0/2148114346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71901501f0 0x7f719014d310 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f71880099a0 tx=0x7f7188019000 comp rx=0 tx=0).stop 2023-10-06T01:41:34.978 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.976+0000 7f71969ae700 1 -- 172.21.15.27:0/2148114346 shutdown_connections 2023-10-06T01:41:34.978 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.976+0000 7f71969ae700 1 --2- 172.21.15.27:0/2148114346 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7170066f30 0x7f71700693e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:34.978 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.976+0000 7f71969ae700 1 --2- 172.21.15.27:0/2148114346 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f71901501f0 0x7f719014d310 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:34.978 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.976+0000 7f71969ae700 1 --2- 172.21.15.27:0/2148114346 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7190068640 0x7f7190144eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:34.978 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.976+0000 7f71969ae700 1 --2- 172.21.15.27:0/2148114346 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f71900aca50 0x7f719014ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:34.978 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.976+0000 7f71969ae700 1 -- 172.21.15.27:0/2148114346 >> 172.21.15.27:0/2148114346 conn(0x7f71900ccd30 msgr2=0x7f719006d5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:34.978 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.976+0000 7f71969ae700 1 -- 172.21.15.27:0/2148114346 shutdown_connections 2023-10-06T01:41:34.978 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:34.976+0000 7f71969ae700 1 -- 172.21.15.27:0/2148114346 wait complete. 2023-10-06T01:41:34.989 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-06T01:41:34.989 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-06T01:41:34.989 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2023-10-06T01:41:34.989 INFO:tasks.cephfs.mount:self.client.name = client.2 2023-10-06T01:41:34.989 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-10-06T01:41:34.989 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-06T01:41:34.990 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-06T01:41:34.990 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-06T01:41:34.990 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2023-10-06T01:41:34.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:34.990 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2023-10-06T01:41:35.005 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-06T01:41:35.005 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-06T01:41:35.005 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2023-10-06T01:41:35.005 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:41:35.006 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2023-10-06T01:41:35.006 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:41:35.006 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-06T01:41:35.006 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2023-10-06T01:41:35.006 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2023-10-06T01:41:35.006 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-06T01:41:35.006 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2023-10-06T01:41:35.006 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2023-10-06T01:41:35.006 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-06T01:41:35.006 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2023-10-06T01:41:35.006 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2023-10-06T01:41:35.006 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-06T01:41:35.007 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 42477sec preferred_lft 42477sec 2023-10-06T01:41:35.007 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link 2023-10-06T01:41:35.007 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:41:35.007 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-06T01:41:35.007 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2023-10-06T01:41:35.007 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2023-10-06T01:41:35.007 INFO:teuthology.orchestra.run.smithi027.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-06T01:41:35.007 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 76:16:3c:5c:63:36 brd ff:ff:ff:ff:ff:ff 2023-10-06T01:41:35.007 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-06T01:41:35.007 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:41:35.007 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::c478:b0ff:fe1c:e43/64 scope link tentative 2023-10-06T01:41:35.007 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:41:35.007 INFO:teuthology.orchestra.run.smithi027.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-06T01:41:35.008 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 76:16:3c:5c:63:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-06T01:41:35.008 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::7416:3cff:fe5c:6336/64 scope link tentative 2023-10-06T01:41:35.008 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:41:35.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:35.008 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-06T01:41:35.068 INFO:teuthology.orchestra.run.smithi027.stdout:1 2023-10-06T01:41:35.069 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:35.070 DEBUG:teuthology.orchestra.run.smithi027:> route 2023-10-06T01:41:35.126 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2023-10-06T01:41:35.126 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-06T01:41:35.127 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-06T01:41:35.127 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-06T01:41:35.127 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-06T01:41:35.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:35.127 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2023-10-06T01:41:35.127 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2023-10-06T01:41:35.127 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-06T01:41:35.127 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-06T01:41:35.127 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-06T01:41:35.127 DEBUG:teuthology.orchestra.run.smithi027:> ') 2023-10-06T01:41:35.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:35.229 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2023-10-06T01:41:35.244 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-06T01:41:35.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:35.245 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list-id 2023-10-06T01:41:35.299 INFO:teuthology.orchestra.run.smithi027.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2023-10-06T01:41:35.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:35.299 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2023-10-06T01:41:35.299 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2023-10-06T01:41:35.300 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-06T01:41:35.300 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2023-10-06T01:41:35.300 DEBUG:teuthology.orchestra.run.smithi027:> ') 2023-10-06T01:41:35.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:35.390 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-10-06T01:41:35.426 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-06T01:41:35.426 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-06T01:41:35.426 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2023-10-06T01:41:35.426 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:41:35.426 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2023-10-06T01:41:35.426 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:41:35.426 INFO:teuthology.orchestra.run.smithi027.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-06T01:41:35.426 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 2a:31:26:ef:a7:62 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-06T01:41:35.427 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-06T01:41:35.427 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:41:35.427 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::2831:26ff:feef:a762/64 scope link tentative 2023-10-06T01:41:35.427 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:41:35.435 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:35.435 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-10-06T01:41:35.511 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-06T01:41:35.511 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-06T01:41:35.511 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2023-10-06T01:41:35.511 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:41:35.512 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2023-10-06T01:41:35.512 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:41:35.512 INFO:teuthology.orchestra.run.smithi027.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-06T01:41:35.512 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 2a:31:26:ef:a7:62 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-06T01:41:35.512 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-06T01:41:35.513 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:41:35.513 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::2831:26ff:feef:a762/64 scope link tentative 2023-10-06T01:41:35.513 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:41:35.519 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2023-10-06T01:41:35.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:35.519 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2023-10-06T01:41:35.519 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2023-10-06T01:41:35.520 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2023-10-06T01:41:35.520 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-10-06T01:41:35.520 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2023-10-06T01:41:35.520 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2023-10-06T01:41:35.520 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2023-10-06T01:41:35.520 DEBUG:teuthology.orchestra.run.smithi027:> ') 2023-10-06T01:41:35.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:35.737 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2023-10-06T01:41:35.737 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2023-10-06T01:41:35.738 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.1 up 2023-10-06T01:41:35.738 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set dev brx.1 master ceph-brx 2023-10-06T01:41:35.738 DEBUG:teuthology.orchestra.run.smithi027:> ') 2023-10-06T01:41:35.819 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T01:41:35.819 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2023-10-06T01:41:35.834 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2023-10-06T01:41:35.834 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T01:41:35.834 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2023-10-06T01:41:35.888 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-06T01:41:35.888 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-10-06T01:41:35.889 DEBUG:teuthology.orchestra.run.smithi027:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2023-10-06T01:41:35.970 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2023-10-06T01:41:35.971 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2023-10-06T01:41:35.971 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2023-10-06T01:41:35.979 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2023-10-06T01:41:35.979 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2023-10-06T01:41:35.979 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2023-10-06T01:41:35.979 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.124:3300,172.21.15.124:3301" 2023-10-06T01:41:35.979 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-06T01:41:36.001 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2023-10-06T01:41:36.001 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2023-10-06T01:41:36.002 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2023-10-06T01:41:36.002 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.124:3300,172.21.15.124:3301" 2023-10-06T01:41:36.002 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-06T01:41:36.002 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:6789,172.21.15.124:6789,172.21.15.124:6790" 2023-10-06T01:41:36.002 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.27:6789,172.21.15.124:6789,172.21.15.124:6790:/ 2023-10-06T01:41:36.002 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs,key=2,fsid=20a7bff0-74f8-4f2f-b6c6-899e804bc965" will pass to kernel 2023-10-06T01:41:36.004 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-06T01:41:36.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:36.004 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2023-10-06T01:41:36.068 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2023-10-06T01:41:36.069 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-06T01:41:36.227 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.226+0000 7f8072bc1700 1 Processor -- start 2023-10-06T01:41:36.228 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.226+0000 7f8072bc1700 1 -- start start 2023-10-06T01:41:36.230 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.227+0000 7f8072bc1700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f806c1571d0 0x7f806c15a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:36.231 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.227+0000 7f8072bc1700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f806c0c79b0 0x7f806c06c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:36.231 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.227+0000 7f8072bc1700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f806c06c6d0 0x7f806c15d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:36.231 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.227+0000 7f8072bc1700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f806c06cb30 con 0x7f806c1571d0 2023-10-06T01:41:36.231 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.227+0000 7f8072bc1700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f806c0cce40 con 0x7f806c0c79b0 2023-10-06T01:41:36.231 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.227+0000 7f8072bc1700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f806c15ac80 con 0x7f806c06c6d0 2023-10-06T01:41:36.231 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.227+0000 7f807095d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f806c1571d0 0x7f806c15a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:36.231 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.227+0000 7f807095d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f806c1571d0 0x7f806c15a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34208/0 (socket says 172.21.15.27:34208) 2023-10-06T01:41:36.232 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.227+0000 7f807095d700 1 -- 172.21.15.27:0/19195107 learned_addr learned my addr 172.21.15.27:0/19195107 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:36.232 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.227+0000 7f806bfff700 1 --2- 172.21.15.27:0/19195107 >> v2:172.21.15.124:3300/0 conn(0x7f806c0c79b0 0x7f806c06c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:36.232 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.227+0000 7f807115e700 1 --2- 172.21.15.27:0/19195107 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f806c06c6d0 0x7f806c15d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:36.232 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.227+0000 7f807095d700 1 -- 172.21.15.27:0/19195107 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f806c06c6d0 msgr2=0x7f806c15d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:36.232 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.227+0000 7f807095d700 1 --2- 172.21.15.27:0/19195107 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f806c06c6d0 0x7f806c15d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:36.232 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.227+0000 7f807095d700 1 -- 172.21.15.27:0/19195107 >> v2:172.21.15.124:3300/0 conn(0x7f806c0c79b0 msgr2=0x7f806c06c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:36.232 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.227+0000 7f807095d700 1 --2- 172.21.15.27:0/19195107 >> v2:172.21.15.124:3300/0 conn(0x7f806c0c79b0 0x7f806c06c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:36.232 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.227+0000 7f807095d700 1 -- 172.21.15.27:0/19195107 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f806c0cbf50 con 0x7f806c1571d0 2023-10-06T01:41:36.233 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.227+0000 7f807095d700 1 --2- 172.21.15.27:0/19195107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f806c1571d0 0x7f806c15a740 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f806c06fc70 tx=0x7f8060009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=185fa4bd7cf2f1ee server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:36.233 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.227+0000 7f806b7fe700 1 -- 172.21.15.27:0/19195107 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f806000b640 con 0x7f806c1571d0 2023-10-06T01:41:36.233 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.227+0000 7f806b7fe700 1 -- 172.21.15.27:0/19195107 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f806000b7a0 con 0x7f806c1571d0 2023-10-06T01:41:36.233 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.227+0000 7f806b7fe700 1 -- 172.21.15.27:0/19195107 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f806000bb60 con 0x7f806c1571d0 2023-10-06T01:41:36.233 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.228+0000 7f8072bc1700 1 -- 172.21.15.27:0/19195107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f806c1571d0 msgr2=0x7f806c15a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:36.233 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.228+0000 7f8072bc1700 1 --2- 172.21.15.27:0/19195107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f806c1571d0 0x7f806c15a740 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f806c06fc70 tx=0x7f8060009250 comp rx=0 tx=0).stop 2023-10-06T01:41:36.234 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.228+0000 7f8072bc1700 1 -- 172.21.15.27:0/19195107 shutdown_connections 2023-10-06T01:41:36.234 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.228+0000 7f8072bc1700 1 --2- 172.21.15.27:0/19195107 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f806c1571d0 0x7f806c15a740 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:36.234 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.228+0000 7f8072bc1700 1 --2- 172.21.15.27:0/19195107 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f806c06c6d0 0x7f806c15d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:36.234 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.228+0000 7f8072bc1700 1 --2- 172.21.15.27:0/19195107 >> v2:172.21.15.124:3300/0 conn(0x7f806c0c79b0 0x7f806c06c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:36.234 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.228+0000 7f8072bc1700 1 -- 172.21.15.27:0/19195107 >> 172.21.15.27:0/19195107 conn(0x7f806c06e6a0 msgr2=0x7f806c06eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:36.234 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.228+0000 7f8072bc1700 1 -- 172.21.15.27:0/19195107 shutdown_connections 2023-10-06T01:41:36.235 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.228+0000 7f8072bc1700 1 -- 172.21.15.27:0/19195107 wait complete. 2023-10-06T01:41:36.235 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.229+0000 7f8072bc1700 1 Processor -- start 2023-10-06T01:41:36.235 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.229+0000 7f8072bc1700 1 -- start start 2023-10-06T01:41:36.237 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.230+0000 7f8072bc1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f806c06c6d0 0x7f806c14b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:36.237 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.230+0000 7f8072bc1700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f806c0c79b0 0x7f806c14a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:36.237 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.230+0000 7f8072bc1700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f806c1571d0 0x7f806c0c69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:36.237 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.230+0000 7f8072bc1700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806c14a740 con 0x7f806c06c6d0 2023-10-06T01:41:36.237 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.230+0000 7f8072bc1700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806c15d2e0 con 0x7f806c0c79b0 2023-10-06T01:41:36.238 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.230+0000 7f8072bc1700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f806c0c44c0 con 0x7f806c1571d0 2023-10-06T01:41:36.238 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.230+0000 7f807095d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f806c06c6d0 0x7f806c14b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:36.238 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.230+0000 7f807095d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f806c06c6d0 0x7f806c14b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34216/0 (socket says 172.21.15.27:34216) 2023-10-06T01:41:36.238 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.230+0000 7f807095d700 1 -- 172.21.15.27:0/3217129145 learned_addr learned my addr 172.21.15.27:0/3217129145 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:36.238 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.230+0000 7f807115e700 1 --2- 172.21.15.27:0/3217129145 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f806c1571d0 0x7f806c0c69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:36.238 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.230+0000 7f806bfff700 1 --2- 172.21.15.27:0/3217129145 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f806c0c79b0 0x7f806c14a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:36.238 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.230+0000 7f807095d700 1 -- 172.21.15.27:0/3217129145 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f806c1571d0 msgr2=0x7f806c0c69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:36.238 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.230+0000 7f807095d700 1 --2- 172.21.15.27:0/3217129145 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f806c1571d0 0x7f806c0c69d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:36.239 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.230+0000 7f807095d700 1 -- 172.21.15.27:0/3217129145 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f806c0c79b0 msgr2=0x7f806c14a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:36.239 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.230+0000 7f807095d700 1 --2- 172.21.15.27:0/3217129145 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f806c0c79b0 0x7f806c14a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:36.239 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.230+0000 7f807095d700 1 -- 172.21.15.27:0/3217129145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80600094f0 con 0x7f806c06c6d0 2023-10-06T01:41:36.239 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.230+0000 7f807115e700 1 --2- 172.21.15.27:0/3217129145 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f806c1571d0 0x7f806c0c69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:41:36.239 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.230+0000 7f806bfff700 1 --2- 172.21.15.27:0/3217129145 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f806c0c79b0 0x7f806c14a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:41:36.239 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.231+0000 7f807095d700 1 --2- 172.21.15.27:0/3217129145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f806c06c6d0 0x7f806c14b120 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f806c067aa0 tx=0x7f8060017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:36.240 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.231+0000 7f8068ff9700 1 -- 172.21.15.27:0/3217129145 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8060000d10 con 0x7f806c06c6d0 2023-10-06T01:41:36.240 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.231+0000 7f8068ff9700 1 -- 172.21.15.27:0/3217129145 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f806000bb20 con 0x7f806c06c6d0 2023-10-06T01:41:36.240 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.231+0000 7f8068ff9700 1 -- 172.21.15.27:0/3217129145 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8060011670 con 0x7f806c06c6d0 2023-10-06T01:41:36.240 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.231+0000 7f8072bc1700 1 -- 172.21.15.27:0/3217129145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f806c0c47a0 con 0x7f806c06c6d0 2023-10-06T01:41:36.240 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.231+0000 7f8072bc1700 1 -- 172.21.15.27:0/3217129145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f806c15d5c0 con 0x7f806c06c6d0 2023-10-06T01:41:36.240 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.232+0000 7f8072bc1700 1 -- 172.21.15.27:0/3217129145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f806c0c4cc0 con 0x7f806c06c6d0 2023-10-06T01:41:36.241 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.232+0000 7f8068ff9700 1 -- 172.21.15.27:0/3217129145 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f80600119e0 con 0x7f806c06c6d0 2023-10-06T01:41:36.241 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.232+0000 7f8068ff9700 1 --2- 172.21.15.27:0/3217129145 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8050066f30 0x7f80500693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:36.241 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.233+0000 7f8068ff9700 1 -- 172.21.15.27:0/3217129145 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f8060085e90 con 0x7f806c06c6d0 2023-10-06T01:41:36.241 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.233+0000 7f806bfff700 1 --2- 172.21.15.27:0/3217129145 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8050066f30 0x7f80500693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:36.241 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.233+0000 7f806bfff700 1 --2- 172.21.15.27:0/3217129145 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8050066f30 0x7f80500693e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8058006f00 tx=0x7f8058008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:36.242 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.237+0000 7f8068ff9700 1 -- 172.21.15.27:0/3217129145 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f806c0c4cc0 con 0x7f806c06c6d0 2023-10-06T01:41:36.480 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.479+0000 7f8072bc1700 1 -- 172.21.15.27:0/3217129145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f806c0c4cc0 con 0x7f806c06c6d0 2023-10-06T01:41:36.481 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.479+0000 7f8068ff9700 1 -- 172.21.15.27:0/3217129145 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7f806c0c4cc0 con 0x7f806c06c6d0 2023-10-06T01:41:36.481 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-06T01:41:36.483 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.481+0000 7f8072bc1700 1 -- 172.21.15.27:0/3217129145 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8050066f30 msgr2=0x7f80500693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:36.483 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.481+0000 7f8072bc1700 1 --2- 172.21.15.27:0/3217129145 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8050066f30 0x7f80500693e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f8058006f00 tx=0x7f8058008040 comp rx=0 tx=0).stop 2023-10-06T01:41:36.483 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.481+0000 7f8072bc1700 1 -- 172.21.15.27:0/3217129145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f806c06c6d0 msgr2=0x7f806c14b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:36.483 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.481+0000 7f8072bc1700 1 --2- 172.21.15.27:0/3217129145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f806c06c6d0 0x7f806c14b120 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f806c067aa0 tx=0x7f8060017000 comp rx=0 tx=0).stop 2023-10-06T01:41:36.483 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.481+0000 7f8072bc1700 1 -- 172.21.15.27:0/3217129145 shutdown_connections 2023-10-06T01:41:36.483 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.481+0000 7f8072bc1700 1 --2- 172.21.15.27:0/3217129145 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8050066f30 0x7f80500693e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:36.483 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.481+0000 7f8072bc1700 1 --2- 172.21.15.27:0/3217129145 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f806c1571d0 0x7f806c0c69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:36.483 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.481+0000 7f8072bc1700 1 --2- 172.21.15.27:0/3217129145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f806c06c6d0 0x7f806c14b120 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:36.483 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.481+0000 7f8072bc1700 1 --2- 172.21.15.27:0/3217129145 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f806c0c79b0 0x7f806c14a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:36.484 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.481+0000 7f8072bc1700 1 -- 172.21.15.27:0/3217129145 >> 172.21.15.27:0/3217129145 conn(0x7f806c06e6a0 msgr2=0x7f806c0c28d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:36.484 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.481+0000 7f8072bc1700 1 -- 172.21.15.27:0/3217129145 shutdown_connections 2023-10-06T01:41:36.484 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:36.481+0000 7f8072bc1700 1 -- 172.21.15.27:0/3217129145 wait complete. 2023-10-06T01:41:36.495 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-06T01:41:36.495 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-06T01:41:36.495 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi124.front.sepia.ceph.com 2023-10-06T01:41:36.495 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-10-06T01:41:36.495 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-10-06T01:41:36.495 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-06T01:41:36.495 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-06T01:41:36.495 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-06T01:41:36.495 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2023-10-06T01:41:36.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:36.496 DEBUG:teuthology.orchestra.run.smithi124:> ip addr 2023-10-06T01:41:36.513 INFO:teuthology.orchestra.run.smithi124.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-06T01:41:36.513 INFO:teuthology.orchestra.run.smithi124.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-06T01:41:36.513 INFO:teuthology.orchestra.run.smithi124.stdout: inet 127.0.0.1/8 scope host lo 2023-10-06T01:41:36.513 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:41:36.513 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 ::1/128 scope host 2023-10-06T01:41:36.514 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:41:36.514 INFO:teuthology.orchestra.run.smithi124.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-06T01:41:36.514 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:d8:a1:a6 brd ff:ff:ff:ff:ff:ff 2023-10-06T01:41:36.514 INFO:teuthology.orchestra.run.smithi124.stdout: altname enp7s0f0 2023-10-06T01:41:36.514 INFO:teuthology.orchestra.run.smithi124.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-06T01:41:36.514 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:d8:a1:a7 brd ff:ff:ff:ff:ff:ff 2023-10-06T01:41:36.514 INFO:teuthology.orchestra.run.smithi124.stdout: altname enp7s0f1 2023-10-06T01:41:36.514 INFO:teuthology.orchestra.run.smithi124.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-06T01:41:36.514 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:8f:d5:c4 brd ff:ff:ff:ff:ff:ff 2023-10-06T01:41:36.514 INFO:teuthology.orchestra.run.smithi124.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-06T01:41:36.514 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:8f:d5:c5 brd ff:ff:ff:ff:ff:ff 2023-10-06T01:41:36.515 INFO:teuthology.orchestra.run.smithi124.stdout: inet 172.21.15.124/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-06T01:41:36.515 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft 42438sec preferred_lft 42438sec 2023-10-06T01:41:36.515 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::ec4:7aff:fe8f:d5c5/64 scope link 2023-10-06T01:41:36.515 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:41:36.515 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-06T01:41:36.515 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:36.515 DEBUG:teuthology.orchestra.run.smithi124:> (cd / && exec bash -c ' 2023-10-06T01:41:36.515 DEBUG:teuthology.orchestra.run.smithi124:> set -e 2023-10-06T01:41:36.515 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link add name ceph-brx type bridge 2023-10-06T01:41:36.516 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip addr flush dev ceph-brx 2023-10-06T01:41:36.516 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link set ceph-brx up 2023-10-06T01:41:36.516 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-06T01:41:36.516 DEBUG:teuthology.orchestra.run.smithi124:> ') 2023-10-06T01:41:36.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:36.686 DEBUG:teuthology.orchestra.run.smithi124:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-06T01:41:36.725 INFO:teuthology.orchestra.run.smithi124.stdout:1 2023-10-06T01:41:36.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:36.727 DEBUG:teuthology.orchestra.run.smithi124:> route 2023-10-06T01:41:36.802 INFO:teuthology.orchestra.run.smithi124.stdout:Kernel IP routing table 2023-10-06T01:41:36.803 INFO:teuthology.orchestra.run.smithi124.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-06T01:41:36.803 INFO:teuthology.orchestra.run.smithi124.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-06T01:41:36.803 INFO:teuthology.orchestra.run.smithi124.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-06T01:41:36.803 INFO:teuthology.orchestra.run.smithi124.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-06T01:41:36.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:36.804 DEBUG:teuthology.orchestra.run.smithi124:> (cd / && exec bash -c ' 2023-10-06T01:41:36.804 DEBUG:teuthology.orchestra.run.smithi124:> set -e 2023-10-06T01:41:36.804 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-06T01:41:36.804 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-06T01:41:36.804 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-06T01:41:36.805 DEBUG:teuthology.orchestra.run.smithi124:> ') 2023-10-06T01:41:37.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:37.026 DEBUG:teuthology.orchestra.run.smithi124:> ip netns list 2023-10-06T01:41:37.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:37.041 DEBUG:teuthology.orchestra.run.smithi124:> ip netns list-id 2023-10-06T01:41:37.095 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:37.096 DEBUG:teuthology.orchestra.run.smithi124:> (cd / && exec bash -c ' 2023-10-06T01:41:37.096 DEBUG:teuthology.orchestra.run.smithi124:> set -e 2023-10-06T01:41:37.096 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-06T01:41:37.096 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2023-10-06T01:41:37.096 DEBUG:teuthology.orchestra.run.smithi124:> ') 2023-10-06T01:41:37.184 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2023-10-06T01:41:37.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:37.185 DEBUG:teuthology.orchestra.run.smithi124:> (cd / && exec bash -c ' 2023-10-06T01:41:37.185 DEBUG:teuthology.orchestra.run.smithi124:> set -e 2023-10-06T01:41:37.185 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2023-10-06T01:41:37.185 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-06T01:41:37.185 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2023-10-06T01:41:37.185 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2023-10-06T01:41:37.185 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2023-10-06T01:41:37.185 DEBUG:teuthology.orchestra.run.smithi124:> ') 2023-10-06T01:41:37.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:37.411 DEBUG:teuthology.orchestra.run.smithi124:> (cd / && exec bash -c ' 2023-10-06T01:41:37.411 DEBUG:teuthology.orchestra.run.smithi124:> set -e 2023-10-06T01:41:37.411 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link set brx.0 up 2023-10-06T01:41:37.411 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link set dev brx.0 master ceph-brx 2023-10-06T01:41:37.411 DEBUG:teuthology.orchestra.run.smithi124:> ') 2023-10-06T01:41:37.503 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T01:41:37.503 DEBUG:teuthology.orchestra.run.smithi124:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-10-06T01:41:37.517 INFO:teuthology.orchestra.run.smithi124.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-10-06T01:41:37.517 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T01:41:37.518 DEBUG:teuthology.orchestra.run.smithi124:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-10-06T01:41:37.572 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-06T01:41:37.572 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-10-06T01:41:37.572 DEBUG:teuthology.orchestra.run.smithi124:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-10-06T01:41:37.659 INFO:teuthology.orchestra.run.smithi124.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-10-06T01:41:37.659 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-10-06T01:41:37.659 INFO:teuthology.orchestra.run.smithi124.stdout:invalid new device string format 2023-10-06T01:41:37.735 INFO:teuthology.orchestra.run.smithi124.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-10-06T01:41:37.735 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-10-06T01:41:37.735 INFO:teuthology.orchestra.run.smithi124.stdout:invalid new device string format 2023-10-06T01:41:37.735 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.124:3300,172.21.15.124:3301" 2023-10-06T01:41:37.735 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-06T01:41:37.757 INFO:teuthology.orchestra.run.smithi124.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-10-06T01:41:37.757 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-10-06T01:41:37.757 INFO:teuthology.orchestra.run.smithi124.stdout:invalid new device string format 2023-10-06T01:41:37.757 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.124:3300,172.21.15.124:3301" 2023-10-06T01:41:37.757 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-06T01:41:37.757 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: resolved to: "172.21.15.27:6789,172.21.15.124:6789,172.21.15.124:6790" 2023-10-06T01:41:37.757 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.27:6789,172.21.15.124:6789,172.21.15.124:6790:/ 2023-10-06T01:41:37.757 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=20a7bff0-74f8-4f2f-b6c6-899e804bc965" will pass to kernel 2023-10-06T01:41:37.759 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-06T01:41:37.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:37.759 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-10-06T01:41:37.783 INFO:tasks.kclient:client.3 config is {'syntax': 'v1'} 2023-10-06T01:41:37.784 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-06T01:41:37.916 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.914+0000 7f5ed87f3700 1 Processor -- start 2023-10-06T01:41:37.916 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.915+0000 7f5ed87f3700 1 -- start start 2023-10-06T01:41:37.918 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.916+0000 7f5ed87f3700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f5ed01596c0 0x7f5ed015d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:37.919 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.916+0000 7f5ed87f3700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5ed00b09a0 0x7f5ed00c8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:37.919 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.916+0000 7f5ed87f3700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5ed015d960 0x7f5ed00c8f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:37.919 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.916+0000 7f5ed87f3700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ed015e3a0 con 0x7f5ed00b09a0 2023-10-06T01:41:37.919 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.916+0000 7f5ed87f3700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ed00cc470 con 0x7f5ed01596c0 2023-10-06T01:41:37.919 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.916+0000 7f5ed87f3700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed015dde0 con 0x7f5ed015d960 2023-10-06T01:41:37.919 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.916+0000 7f5ed5d8e700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5ed00b09a0 0x7f5ed00c8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:37.919 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.916+0000 7f5ed5d8e700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5ed00b09a0 0x7f5ed00c8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34220/0 (socket says 172.21.15.27:34220) 2023-10-06T01:41:37.919 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.916+0000 7f5ed5d8e700 1 -- 172.21.15.27:0/632241058 learned_addr learned my addr 172.21.15.27:0/632241058 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:37.919 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.916+0000 7f5ed6d90700 1 --2- 172.21.15.27:0/632241058 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5ed015d960 0x7f5ed00c8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:37.919 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.916+0000 7f5ed658f700 1 --2- 172.21.15.27:0/632241058 >> v2:172.21.15.124:3300/0 conn(0x7f5ed01596c0 0x7f5ed015d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:37.919 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.916+0000 7f5ed5d8e700 1 -- 172.21.15.27:0/632241058 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5ed015d960 msgr2=0x7f5ed00c8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:37.920 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.916+0000 7f5ed5d8e700 1 --2- 172.21.15.27:0/632241058 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5ed015d960 0x7f5ed00c8f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:37.920 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.916+0000 7f5ed5d8e700 1 -- 172.21.15.27:0/632241058 >> v2:172.21.15.124:3300/0 conn(0x7f5ed01596c0 msgr2=0x7f5ed015d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:37.920 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.916+0000 7f5ed5d8e700 1 --2- 172.21.15.27:0/632241058 >> v2:172.21.15.124:3300/0 conn(0x7f5ed01596c0 0x7f5ed015d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:37.920 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.916+0000 7f5ed5d8e700 1 -- 172.21.15.27:0/632241058 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed015e0c0 con 0x7f5ed00b09a0 2023-10-06T01:41:37.920 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.916+0000 7f5ed5d8e700 1 --2- 172.21.15.27:0/632241058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ed00b09a0 0x7f5ed00c8a40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5ed006f4a0 tx=0x7f5ec4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f1152bbd85acec server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:37.920 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.916+0000 7f5ed558d700 1 -- 172.21.15.27:0/632241058 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec400b640 con 0x7f5ed00b09a0 2023-10-06T01:41:37.920 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.917+0000 7f5ed558d700 1 -- 172.21.15.27:0/632241058 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ec400b7a0 con 0x7f5ed00b09a0 2023-10-06T01:41:37.920 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.917+0000 7f5ed558d700 1 -- 172.21.15.27:0/632241058 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec400bb60 con 0x7f5ed00b09a0 2023-10-06T01:41:37.920 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.917+0000 7f5ed87f3700 1 -- 172.21.15.27:0/632241058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ed00b09a0 msgr2=0x7f5ed00c8a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:37.920 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.917+0000 7f5ed87f3700 1 --2- 172.21.15.27:0/632241058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ed00b09a0 0x7f5ed00c8a40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f5ed006f4a0 tx=0x7f5ec4009250 comp rx=0 tx=0).stop 2023-10-06T01:41:37.920 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.917+0000 7f5ed87f3700 1 -- 172.21.15.27:0/632241058 shutdown_connections 2023-10-06T01:41:37.921 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.917+0000 7f5ed87f3700 1 --2- 172.21.15.27:0/632241058 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5ed015d960 0x7f5ed00c8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:37.921 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.917+0000 7f5ed87f3700 1 --2- 172.21.15.27:0/632241058 >> v2:172.21.15.124:3300/0 conn(0x7f5ed01596c0 0x7f5ed015d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:37.921 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.917+0000 7f5ed87f3700 1 --2- 172.21.15.27:0/632241058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ed00b09a0 0x7f5ed00c8a40 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:37.921 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.917+0000 7f5ed87f3700 1 -- 172.21.15.27:0/632241058 >> 172.21.15.27:0/632241058 conn(0x7f5ed0067630 msgr2=0x7f5ed0067a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:37.921 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.917+0000 7f5ed87f3700 1 -- 172.21.15.27:0/632241058 shutdown_connections 2023-10-06T01:41:37.921 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.917+0000 7f5ed87f3700 1 -- 172.21.15.27:0/632241058 wait complete. 2023-10-06T01:41:37.923 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.918+0000 7f5ed87f3700 1 Processor -- start 2023-10-06T01:41:37.924 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.918+0000 7f5ed87f3700 1 -- start start 2023-10-06T01:41:37.924 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.919+0000 7f5ed87f3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ed01596c0 0x7f5ed0158af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:37.924 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.919+0000 7f5ed87f3700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5ed00c4840 0x7f5ed00c2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:37.924 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.919+0000 7f5ed87f3700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5ed00c7010 0x7f5ed00c7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:37.924 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.919+0000 7f5ed87f3700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed00c79c0 con 0x7f5ed01596c0 2023-10-06T01:41:37.924 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.919+0000 7f5ed87f3700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed015a8f0 con 0x7f5ed00c7010 2023-10-06T01:41:37.924 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.919+0000 7f5ed87f3700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed00c2f20 con 0x7f5ed00c4840 2023-10-06T01:41:37.924 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.919+0000 7f5ed658f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ed01596c0 0x7f5ed0158af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:37.924 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.919+0000 7f5ed658f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ed01596c0 0x7f5ed0158af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34228/0 (socket says 172.21.15.27:34228) 2023-10-06T01:41:37.924 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.919+0000 7f5ed658f700 1 -- 172.21.15.27:0/2264096132 learned_addr learned my addr 172.21.15.27:0/2264096132 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:37.925 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.919+0000 7f5ed5d8e700 1 --2- 172.21.15.27:0/2264096132 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5ed00c4840 0x7f5ed00c2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:37.925 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.919+0000 7f5ed6d90700 1 --2- 172.21.15.27:0/2264096132 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5ed00c7010 0x7f5ed00c7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:37.925 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.919+0000 7f5ed658f700 1 -- 172.21.15.27:0/2264096132 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5ed00c4840 msgr2=0x7f5ed00c2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:37.925 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.919+0000 7f5ed658f700 1 --2- 172.21.15.27:0/2264096132 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5ed00c4840 0x7f5ed00c2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:37.925 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.919+0000 7f5ed658f700 1 -- 172.21.15.27:0/2264096132 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5ed00c7010 msgr2=0x7f5ed00c7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:37.925 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.919+0000 7f5ed658f700 1 --2- 172.21.15.27:0/2264096132 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5ed00c7010 0x7f5ed00c7480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:37.925 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.919+0000 7f5ed658f700 1 -- 172.21.15.27:0/2264096132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ec40094f0 con 0x7f5ed01596c0 2023-10-06T01:41:37.925 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.920+0000 7f5ed658f700 1 --2- 172.21.15.27:0/2264096132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ed01596c0 0x7f5ed0158af0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5ec800ec80 tx=0x7f5ec80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:37.925 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.920+0000 7f5ece7fc700 1 -- 172.21.15.27:0/2264096132 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec800c9d0 con 0x7f5ed01596c0 2023-10-06T01:41:37.925 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.920+0000 7f5ece7fc700 1 -- 172.21.15.27:0/2264096132 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ec800cb30 con 0x7f5ed01596c0 2023-10-06T01:41:37.925 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.920+0000 7f5ed87f3700 1 -- 172.21.15.27:0/2264096132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ed00c3200 con 0x7f5ed01596c0 2023-10-06T01:41:37.926 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.920+0000 7f5ece7fc700 1 -- 172.21.15.27:0/2264096132 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec8013470 con 0x7f5ed01596c0 2023-10-06T01:41:37.926 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.920+0000 7f5ed87f3700 1 -- 172.21.15.27:0/2264096132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ed015a510 con 0x7f5ed01596c0 2023-10-06T01:41:37.926 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.921+0000 7f5ed87f3700 1 -- 172.21.15.27:0/2264096132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ed0064050 con 0x7f5ed01596c0 2023-10-06T01:41:37.926 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.921+0000 7f5ece7fc700 1 -- 172.21.15.27:0/2264096132 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5ec8013820 con 0x7f5ed01596c0 2023-10-06T01:41:37.926 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.922+0000 7f5ece7fc700 1 --2- 172.21.15.27:0/2264096132 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5ebc066f30 0x7f5ebc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:37.926 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.922+0000 7f5ece7fc700 1 -- 172.21.15.27:0/2264096132 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f5ec8084a80 con 0x7f5ed01596c0 2023-10-06T01:41:37.926 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.922+0000 7f5ed5d8e700 1 --2- 172.21.15.27:0/2264096132 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5ebc066f30 0x7f5ebc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:37.926 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.922+0000 7f5ed5d8e700 1 --2- 172.21.15.27:0/2264096132 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5ebc066f30 0x7f5ebc0693e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5ed00b4580 tx=0x7f5ec4017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:37.927 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:37.926+0000 7f5ece7fc700 1 -- 172.21.15.27:0/2264096132 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5ec8051b90 con 0x7f5ed01596c0 2023-10-06T01:41:38.168 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:38.167+0000 7f5ed87f3700 1 -- 172.21.15.27:0/2264096132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f5ed0159030 con 0x7f5ed01596c0 2023-10-06T01:41:38.168 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:38.167+0000 7f5ece7fc700 1 -- 172.21.15.27:0/2264096132 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7f5ec80551b0 con 0x7f5ed01596c0 2023-10-06T01:41:38.168 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-06T01:41:38.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:38.169+0000 7f5ed87f3700 1 -- 172.21.15.27:0/2264096132 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5ebc066f30 msgr2=0x7f5ebc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:38.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:38.169+0000 7f5ed87f3700 1 --2- 172.21.15.27:0/2264096132 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5ebc066f30 0x7f5ebc0693e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5ed00b4580 tx=0x7f5ec4017000 comp rx=0 tx=0).stop 2023-10-06T01:41:38.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:38.169+0000 7f5ed87f3700 1 -- 172.21.15.27:0/2264096132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ed01596c0 msgr2=0x7f5ed0158af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:38.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:38.169+0000 7f5ed87f3700 1 --2- 172.21.15.27:0/2264096132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ed01596c0 0x7f5ed0158af0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5ec800ec80 tx=0x7f5ec80091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:38.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:38.169+0000 7f5ed87f3700 1 -- 172.21.15.27:0/2264096132 shutdown_connections 2023-10-06T01:41:38.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:38.169+0000 7f5ed87f3700 1 --2- 172.21.15.27:0/2264096132 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5ebc066f30 0x7f5ebc0693e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:38.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:38.169+0000 7f5ed87f3700 1 --2- 172.21.15.27:0/2264096132 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5ed00c7010 0x7f5ed00c7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:38.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:38.169+0000 7f5ed87f3700 1 --2- 172.21.15.27:0/2264096132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5ed01596c0 0x7f5ed0158af0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:38.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:38.169+0000 7f5ed87f3700 1 --2- 172.21.15.27:0/2264096132 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5ed00c4840 0x7f5ed00c2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:38.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:38.169+0000 7f5ed87f3700 1 -- 172.21.15.27:0/2264096132 >> 172.21.15.27:0/2264096132 conn(0x7f5ed0067630 msgr2=0x7f5ed0068f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:38.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:38.169+0000 7f5ed87f3700 1 -- 172.21.15.27:0/2264096132 shutdown_connections 2023-10-06T01:41:38.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:38.169+0000 7f5ed87f3700 1 -- 172.21.15.27:0/2264096132 wait complete. 2023-10-06T01:41:38.182 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-06T01:41:38.182 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-06T01:41:38.182 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi124.front.sepia.ceph.com 2023-10-06T01:41:38.182 INFO:tasks.cephfs.mount:self.client.name = client.3 2023-10-06T01:41:38.182 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-10-06T01:41:38.183 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-06T01:41:38.183 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-06T01:41:38.183 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-06T01:41:38.183 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2023-10-06T01:41:38.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:38.183 DEBUG:teuthology.orchestra.run.smithi124:> ip addr 2023-10-06T01:41:38.199 INFO:teuthology.orchestra.run.smithi124.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-06T01:41:38.199 INFO:teuthology.orchestra.run.smithi124.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-06T01:41:38.199 INFO:teuthology.orchestra.run.smithi124.stdout: inet 127.0.0.1/8 scope host lo 2023-10-06T01:41:38.199 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:41:38.199 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 ::1/128 scope host 2023-10-06T01:41:38.199 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:41:38.199 INFO:teuthology.orchestra.run.smithi124.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-06T01:41:38.199 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:d8:a1:a6 brd ff:ff:ff:ff:ff:ff 2023-10-06T01:41:38.199 INFO:teuthology.orchestra.run.smithi124.stdout: altname enp7s0f0 2023-10-06T01:41:38.199 INFO:teuthology.orchestra.run.smithi124.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-06T01:41:38.200 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:d8:a1:a7 brd ff:ff:ff:ff:ff:ff 2023-10-06T01:41:38.200 INFO:teuthology.orchestra.run.smithi124.stdout: altname enp7s0f1 2023-10-06T01:41:38.200 INFO:teuthology.orchestra.run.smithi124.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-06T01:41:38.200 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:8f:d5:c4 brd ff:ff:ff:ff:ff:ff 2023-10-06T01:41:38.200 INFO:teuthology.orchestra.run.smithi124.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-06T01:41:38.200 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 0c:c4:7a:8f:d5:c5 brd ff:ff:ff:ff:ff:ff 2023-10-06T01:41:38.200 INFO:teuthology.orchestra.run.smithi124.stdout: inet 172.21.15.124/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-06T01:41:38.200 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft 42436sec preferred_lft 42436sec 2023-10-06T01:41:38.200 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::ec4:7aff:fe8f:d5c5/64 scope link 2023-10-06T01:41:38.200 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:41:38.200 INFO:teuthology.orchestra.run.smithi124.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-06T01:41:38.200 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 76:13:10:ce:ac:77 brd ff:ff:ff:ff:ff:ff 2023-10-06T01:41:38.200 INFO:teuthology.orchestra.run.smithi124.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-06T01:41:38.201 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:41:38.201 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::ec6b:e1ff:fe24:a0fe/64 scope link tentative 2023-10-06T01:41:38.201 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:41:38.201 INFO:teuthology.orchestra.run.smithi124.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-06T01:41:38.201 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether 76:13:10:ce:ac:77 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-06T01:41:38.201 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::7413:10ff:fece:ac77/64 scope link tentative 2023-10-06T01:41:38.201 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:41:38.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:38.201 DEBUG:teuthology.orchestra.run.smithi124:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-06T01:41:38.263 INFO:teuthology.orchestra.run.smithi124.stdout:1 2023-10-06T01:41:38.264 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:38.265 DEBUG:teuthology.orchestra.run.smithi124:> route 2023-10-06T01:41:38.321 INFO:teuthology.orchestra.run.smithi124.stdout:Kernel IP routing table 2023-10-06T01:41:38.322 INFO:teuthology.orchestra.run.smithi124.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-06T01:41:38.322 INFO:teuthology.orchestra.run.smithi124.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-06T01:41:38.322 INFO:teuthology.orchestra.run.smithi124.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-06T01:41:38.322 INFO:teuthology.orchestra.run.smithi124.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-06T01:41:38.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:38.322 DEBUG:teuthology.orchestra.run.smithi124:> (cd / && exec bash -c ' 2023-10-06T01:41:38.322 DEBUG:teuthology.orchestra.run.smithi124:> set -e 2023-10-06T01:41:38.322 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-06T01:41:38.323 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-06T01:41:38.323 DEBUG:teuthology.orchestra.run.smithi124:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-06T01:41:38.323 DEBUG:teuthology.orchestra.run.smithi124:> ') 2023-10-06T01:41:38.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:38.422 DEBUG:teuthology.orchestra.run.smithi124:> ip netns list 2023-10-06T01:41:38.436 INFO:teuthology.orchestra.run.smithi124.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-10-06T01:41:38.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:38.437 DEBUG:teuthology.orchestra.run.smithi124:> ip netns list-id 2023-10-06T01:41:38.492 INFO:teuthology.orchestra.run.smithi124.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2023-10-06T01:41:38.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:38.492 DEBUG:teuthology.orchestra.run.smithi124:> (cd / && exec bash -c ' 2023-10-06T01:41:38.492 DEBUG:teuthology.orchestra.run.smithi124:> set -e 2023-10-06T01:41:38.492 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2023-10-06T01:41:38.493 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2023-10-06T01:41:38.493 DEBUG:teuthology.orchestra.run.smithi124:> ') 2023-10-06T01:41:38.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:38.580 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-10-06T01:41:38.620 INFO:teuthology.orchestra.run.smithi124.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-06T01:41:38.620 INFO:teuthology.orchestra.run.smithi124.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-06T01:41:38.620 INFO:teuthology.orchestra.run.smithi124.stdout: inet 127.0.0.1/8 scope host lo 2023-10-06T01:41:38.620 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:41:38.620 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 ::1/128 scope host 2023-10-06T01:41:38.620 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:41:38.620 INFO:teuthology.orchestra.run.smithi124.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-06T01:41:38.620 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether d6:e6:ff:53:85:0c brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-06T01:41:38.620 INFO:teuthology.orchestra.run.smithi124.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-06T01:41:38.620 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:41:38.620 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::d4e6:ffff:fe53:850c/64 scope link tentative 2023-10-06T01:41:38.621 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:41:38.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:38.633 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-10-06T01:41:38.713 INFO:teuthology.orchestra.run.smithi124.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-06T01:41:38.713 INFO:teuthology.orchestra.run.smithi124.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-06T01:41:38.713 INFO:teuthology.orchestra.run.smithi124.stdout: inet 127.0.0.1/8 scope host lo 2023-10-06T01:41:38.713 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:41:38.713 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 ::1/128 scope host 2023-10-06T01:41:38.713 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:41:38.713 INFO:teuthology.orchestra.run.smithi124.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-06T01:41:38.713 INFO:teuthology.orchestra.run.smithi124.stdout: link/ether d6:e6:ff:53:85:0c brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-06T01:41:38.713 INFO:teuthology.orchestra.run.smithi124.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-06T01:41:38.713 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:41:38.714 INFO:teuthology.orchestra.run.smithi124.stdout: inet6 fe80::d4e6:ffff:fe53:850c/64 scope link tentative 2023-10-06T01:41:38.714 INFO:teuthology.orchestra.run.smithi124.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:41:38.726 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2023-10-06T01:41:38.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:38.727 DEBUG:teuthology.orchestra.run.smithi124:> (cd / && exec bash -c ' 2023-10-06T01:41:38.727 DEBUG:teuthology.orchestra.run.smithi124:> set -e 2023-10-06T01:41:38.727 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2023-10-06T01:41:38.727 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-10-06T01:41:38.727 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2023-10-06T01:41:38.727 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2023-10-06T01:41:38.727 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2023-10-06T01:41:38.727 DEBUG:teuthology.orchestra.run.smithi124:> ') 2023-10-06T01:41:38.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:38.934 DEBUG:teuthology.orchestra.run.smithi124:> (cd / && exec bash -c ' 2023-10-06T01:41:38.934 DEBUG:teuthology.orchestra.run.smithi124:> set -e 2023-10-06T01:41:38.934 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link set brx.1 up 2023-10-06T01:41:38.934 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link set dev brx.1 master ceph-brx 2023-10-06T01:41:38.934 DEBUG:teuthology.orchestra.run.smithi124:> ') 2023-10-06T01:41:39.013 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T01:41:39.014 DEBUG:teuthology.orchestra.run.smithi124:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2023-10-06T01:41:39.028 INFO:teuthology.orchestra.run.smithi124.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2023-10-06T01:41:39.028 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T01:41:39.028 DEBUG:teuthology.orchestra.run.smithi124:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2023-10-06T01:41:39.082 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-06T01:41:39.082 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-10-06T01:41:39.082 DEBUG:teuthology.orchestra.run.smithi124:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2023-10-06T01:41:39.161 INFO:teuthology.orchestra.run.smithi124.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2023-10-06T01:41:39.162 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2023-10-06T01:41:39.162 INFO:teuthology.orchestra.run.smithi124.stdout:invalid new device string format 2023-10-06T01:41:39.170 INFO:teuthology.orchestra.run.smithi124.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2023-10-06T01:41:39.170 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2023-10-06T01:41:39.170 INFO:teuthology.orchestra.run.smithi124.stdout:invalid new device string format 2023-10-06T01:41:39.170 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.124:3300,172.21.15.124:3301" 2023-10-06T01:41:39.170 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-06T01:41:39.191 INFO:teuthology.orchestra.run.smithi124.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2023-10-06T01:41:39.192 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2023-10-06T01:41:39.192 INFO:teuthology.orchestra.run.smithi124.stdout:invalid new device string format 2023-10-06T01:41:39.192 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.124:3300,172.21.15.124:3301" 2023-10-06T01:41:39.192 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-06T01:41:39.192 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: resolved to: "172.21.15.27:6789,172.21.15.124:6789,172.21.15.124:6790" 2023-10-06T01:41:39.192 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.27:6789,172.21.15.124:6789,172.21.15.124:6790:/ 2023-10-06T01:41:39.192 INFO:teuthology.orchestra.run.smithi124.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs,key=3,fsid=20a7bff0-74f8-4f2f-b6c6-899e804bc965" will pass to kernel 2023-10-06T01:41:39.193 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-06T01:41:39.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:39.194 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2023-10-06T01:41:39.258 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2023-10-06T01:41:39.277 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T01:41:39.408 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.407+0000 7f0c529e7700 1 Processor -- start 2023-10-06T01:41:39.408 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.407+0000 7f0c529e7700 1 -- start start 2023-10-06T01:41:39.410 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.408+0000 7f0c529e7700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0c4c15d850 0x7f0c4c0b5c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:39.410 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.408+0000 7f0c529e7700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f0c4c0b6170 0x7f0c4c0b0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:39.410 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.408+0000 7f0c529e7700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c4c069f70 con 0x7f0c4c15d480 2023-10-06T01:41:39.410 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.408+0000 7f0c529e7700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c4c0687c0 con 0x7f0c4c0b6170 2023-10-06T01:41:39.410 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.408+0000 7f0c529e7700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4c068aa0 con 0x7f0c4c15d850 2023-10-06T01:41:39.411 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.408+0000 7f0c519e5700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0c4c15d480 0x7f0c4c069860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40996/0 (socket says 172.21.15.27:40996) 2023-10-06T01:41:39.411 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.408+0000 7f0c519e5700 1 -- 172.21.15.27:0/2347696041 learned_addr learned my addr 172.21.15.27:0/2347696041 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:39.411 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.408+0000 7f0c511e4700 1 --2- 172.21.15.27:0/2347696041 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0c4c15d850 0x7f0c4c0b5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:39.411 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.408+0000 7f0c521e6700 1 --2- 172.21.15.27:0/2347696041 >> v2:172.21.15.124:3300/0 conn(0x7f0c4c0b6170 0x7f0c4c0b0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:39.411 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.408+0000 7f0c509e3700 1 -- 172.21.15.27:0/2347696041 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2616947320 0 0) 0x7f0c4c069f70 con 0x7f0c4c15d480 2023-10-06T01:41:39.411 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.408+0000 7f0c509e3700 1 -- 172.21.15.27:0/2347696041 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c34003570 con 0x7f0c4c15d480 2023-10-06T01:41:39.411 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.408+0000 7f0c521e6700 1 -- 172.21.15.27:0/2347696041 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0c4c15d850 msgr2=0x7f0c4c0b5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:39.411 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.408+0000 7f0c521e6700 1 --2- 172.21.15.27:0/2347696041 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0c4c15d850 0x7f0c4c0b5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:39.411 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.408+0000 7f0c521e6700 1 -- 172.21.15.27:0/2347696041 >> v1:172.21.15.27:6789/0 conn(0x7f0c4c15d480 legacy=0x7f0c4c069860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:39.411 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.408+0000 7f0c521e6700 1 -- 172.21.15.27:0/2347696041 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c4c068d80 con 0x7f0c4c0b6170 2023-10-06T01:41:39.411 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.409+0000 7f0c521e6700 1 --2- 172.21.15.27:0/2347696041 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c4c0b6170 0x7f0c4c0b0310 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c0098e0 tx=0x7f0c3c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c04ea720564504ff server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:39.411 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.409+0000 7f0c509e3700 1 -- 172.21.15.27:0/2347696041 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3c011630 con 0x7f0c4c0b6170 2023-10-06T01:41:39.412 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.409+0000 7f0c509e3700 1 -- 172.21.15.27:0/2347696041 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c3c011790 con 0x7f0c4c0b6170 2023-10-06T01:41:39.412 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.409+0000 7f0c509e3700 1 -- 172.21.15.27:0/2347696041 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3c011b30 con 0x7f0c4c0b6170 2023-10-06T01:41:39.414 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.409+0000 7f0c529e7700 1 -- 172.21.15.27:0/2347696041 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c4c0b6170 msgr2=0x7f0c4c0b0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:39.414 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.409+0000 7f0c529e7700 1 --2- 172.21.15.27:0/2347696041 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c4c0b6170 0x7f0c4c0b0310 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c0098e0 tx=0x7f0c3c0091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:39.415 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.409+0000 7f0c529e7700 1 -- 172.21.15.27:0/2347696041 shutdown_connections 2023-10-06T01:41:39.415 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.409+0000 7f0c529e7700 1 --2- 172.21.15.27:0/2347696041 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c4c0b6170 0x7f0c4c0b0310 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:39.415 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.409+0000 7f0c529e7700 1 --2- 172.21.15.27:0/2347696041 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0c4c15d850 0x7f0c4c0b5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:39.415 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.410+0000 7f0c529e7700 1 -- 172.21.15.27:0/2347696041 >> 172.21.15.27:0/2347696041 conn(0x7f0c4c0ccf80 msgr2=0x7f0c4c0cd380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:39.415 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.410+0000 7f0c529e7700 1 -- 172.21.15.27:0/2347696041 shutdown_connections 2023-10-06T01:41:39.415 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.410+0000 7f0c529e7700 1 -- 172.21.15.27:0/2347696041 wait complete. 2023-10-06T01:41:39.415 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.411+0000 7f0c529e7700 1 Processor -- start 2023-10-06T01:41:39.415 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.411+0000 7f0c529e7700 1 -- start start 2023-10-06T01:41:39.415 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.412+0000 7f0c529e7700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c4c0b6170 0x7f0c4c158bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:39.415 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.412+0000 7f0c529e7700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c4c15d480 0x7f0c4c1419a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:39.415 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.412+0000 7f0c529e7700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0c4c15d850 0x7f0c4c141ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:39.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.412+0000 7f0c529e7700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4c140660 con 0x7f0c4c0b6170 2023-10-06T01:41:39.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.412+0000 7f0c529e7700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4c16d9d0 con 0x7f0c4c15d480 2023-10-06T01:41:39.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.412+0000 7f0c529e7700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4c0c8210 con 0x7f0c4c15d850 2023-10-06T01:41:39.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.412+0000 7f0c519e5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c4c0b6170 0x7f0c4c158bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:39.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.412+0000 7f0c519e5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c4c0b6170 0x7f0c4c158bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34242/0 (socket says 172.21.15.27:34242) 2023-10-06T01:41:39.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.412+0000 7f0c519e5700 1 -- 172.21.15.27:0/3111177789 learned_addr learned my addr 172.21.15.27:0/3111177789 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:39.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.412+0000 7f0c511e4700 1 --2- 172.21.15.27:0/3111177789 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c4c15d480 0x7f0c4c1419a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:39.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.412+0000 7f0c521e6700 1 --2- 172.21.15.27:0/3111177789 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0c4c15d850 0x7f0c4c141ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:39.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.412+0000 7f0c519e5700 1 -- 172.21.15.27:0/3111177789 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0c4c15d850 msgr2=0x7f0c4c141ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:39.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.412+0000 7f0c519e5700 1 --2- 172.21.15.27:0/3111177789 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0c4c15d850 0x7f0c4c141ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:39.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.412+0000 7f0c519e5700 1 -- 172.21.15.27:0/3111177789 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c4c15d480 msgr2=0x7f0c4c1419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:39.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.412+0000 7f0c519e5700 1 --2- 172.21.15.27:0/3111177789 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c4c15d480 0x7f0c4c1419a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:39.417 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.412+0000 7f0c519e5700 1 -- 172.21.15.27:0/3111177789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c3c009420 con 0x7f0c4c0b6170 2023-10-06T01:41:39.417 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.412+0000 7f0c511e4700 1 --2- 172.21.15.27:0/3111177789 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c4c15d480 0x7f0c4c1419a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:41:39.417 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.412+0000 7f0c519e5700 1 --2- 172.21.15.27:0/3111177789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c4c0b6170 0x7f0c4c158bd0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0c4400cd60 tx=0x7f0c44001b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:39.417 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.412+0000 7f0c49ffb700 1 -- 172.21.15.27:0/3111177789 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4400aa80 con 0x7f0c4c0b6170 2023-10-06T01:41:39.417 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.412+0000 7f0c49ffb700 1 -- 172.21.15.27:0/3111177789 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c4400ac00 con 0x7f0c4c0b6170 2023-10-06T01:41:39.417 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.412+0000 7f0c529e7700 1 -- 172.21.15.27:0/3111177789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c4c16d6f0 con 0x7f0c4c0b6170 2023-10-06T01:41:39.417 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.412+0000 7f0c49ffb700 1 -- 172.21.15.27:0/3111177789 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c440070f0 con 0x7f0c4c0b6170 2023-10-06T01:41:39.417 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.413+0000 7f0c529e7700 1 -- 172.21.15.27:0/3111177789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c4c0c7ea0 con 0x7f0c4c0b6170 2023-10-06T01:41:39.418 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.414+0000 7f0c529e7700 1 -- 172.21.15.27:0/3111177789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c4c04c3c0 con 0x7f0c4c0b6170 2023-10-06T01:41:39.418 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.414+0000 7f0c49ffb700 1 -- 172.21.15.27:0/3111177789 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0c440074a0 con 0x7f0c4c0b6170 2023-10-06T01:41:39.418 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.414+0000 7f0c49ffb700 1 --2- 172.21.15.27:0/3111177789 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0c38066f30 0x7f0c380693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:39.418 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.415+0000 7f0c49ffb700 1 -- 172.21.15.27:0/3111177789 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f0c44084a80 con 0x7f0c4c0b6170 2023-10-06T01:41:39.418 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.415+0000 7f0c511e4700 1 --2- 172.21.15.27:0/3111177789 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0c38066f30 0x7f0c380693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:39.419 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.415+0000 7f0c511e4700 1 --2- 172.21.15.27:0/3111177789 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0c38066f30 0x7f0c380693e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c06fbb0 tx=0x7f0c400091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:39.420 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.419+0000 7f0c49ffb700 1 -- 172.21.15.27:0/3111177789 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0c44051c10 con 0x7f0c4c0b6170 2023-10-06T01:41:39.615 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.614+0000 7f0c529e7700 1 -- 172.21.15.27:0/3111177789 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c4c069060 con 0x7f0c4c0b6170 2023-10-06T01:41:39.616 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.615+0000 7f0c49ffb700 1 -- 172.21.15.27:0/3111177789 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16866 (secure 0 0 0) 0x7f0c44055230 con 0x7f0c4c0b6170 2023-10-06T01:41:39.617 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:39.617 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":20,"fsid":"20a7bff0-74f8-4f2f-b6c6-899e804bc965","created":"2023-10-06T01:40:54.474215+0000","modified":"2023-10-06T01:41:14.324546+0000","last_up_change":"2023-10-06T01:41:05.242847+0000","last_in_change":"2023-10-06T01:40:56.890960+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-10-06T01:41:05.685386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-10-06T01:41:06.092995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T01:41:10.434257+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T01:41:11.700262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b902c86d-800c-40c5-b152-b545e50d02c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":90644},{"type":"v1","addr":"172.21.15.27:6801","nonce":90644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":90644},{"type":"v1","addr":"172.21.15.27:6803","nonce":90644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":90644},{"type":"v1","addr":"172.21.15.27:6807","nonce":90644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":90644},{"type":"v1","addr":"172.21.15.27:6805","nonce":90644}]},"public_addr":"172.21.15.27:6801/90644","cluster_addr":"172.21.15.27:6803/90644","heartbeat_back_addr":"172.21.15.27:6807/90644","heartbeat_front_addr":"172.21.15.27:6805/90644","state":["exists","up"]},{"osd":1,"uuid":"5b4a70e5-b06e-48cd-bfc4-7962c5d05386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":90643},{"type":"v1","addr":"172.21.15.27:6809","nonce":90643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":90643},{"type":"v1","addr":"172.21.15.27:6811","nonce":90643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":90643},{"type":"v1","addr":"172.21.15.27:6815","nonce":90643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":90643},{"type":"v1","addr":"172.21.15.27:6813","nonce":90643}]},"public_addr":"172.21.15.27:6809/90643","cluster_addr":"172.21.15.27:6811/90643","heartbeat_back_addr":"172.21.15.27:6815/90643","heartbeat_front_addr":"172.21.15.27:6813/90643","state":["exists","up"]},{"osd":2,"uuid":"22d9de46-4073-4336-9122-5227e601eca0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":90645},{"type":"v1","addr":"172.21.15.27:6817","nonce":90645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":90645},{"type":"v1","addr":"172.21.15.27:6819","nonce":90645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":90645},{"type":"v1","addr":"172.21.15.27:6823","nonce":90645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":90645},{"type":"v1","addr":"172.21.15.27:6821","nonce":90645}]},"public_addr":"172.21.15.27:6817/90645","cluster_addr":"172.21.15.27:6819/90645","heartbeat_back_addr":"172.21.15.27:6823/90645","heartbeat_front_addr":"172.21.15.27:6821/90645","state":["exists","up"]},{"osd":3,"uuid":"438bb2d3-34e2-4b7c-9734-8360c1b67650","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":90668},{"type":"v1","addr":"172.21.15.27:6825","nonce":90668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":90668},{"type":"v1","addr":"172.21.15.27:6827","nonce":90668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":90668},{"type":"v1","addr":"172.21.15.27:6831","nonce":90668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":90668},{"type":"v1","addr":"172.21.15.27:6829","nonce":90668}]},"public_addr":"172.21.15.27:6825/90668","cluster_addr":"172.21.15.27:6827/90668","heartbeat_back_addr":"172.21.15.27:6831/90668","heartbeat_front_addr":"172.21.15.27:6829/90668","state":["exists","up"]},{"osd":4,"uuid":"e1956a8e-e0f8-4e7c-acbb-40037415a354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":90863},{"type":"v1","addr":"172.21.15.124:6801","nonce":90863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":90863},{"type":"v1","addr":"172.21.15.124:6803","nonce":90863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":90863},{"type":"v1","addr":"172.21.15.124:6807","nonce":90863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":90863},{"type":"v1","addr":"172.21.15.124:6805","nonce":90863}]},"public_addr":"172.21.15.124:6801/90863","cluster_addr":"172.21.15.124:6803/90863","heartbeat_back_addr":"172.21.15.124:6807/90863","heartbeat_front_addr":"172.21.15.124:6805/90863","state":["exists","up"]},{"osd":5,"uuid":"306c59d4-846d-4b73-b1c9-051bec9e9368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":90885},{"type":"v1","addr":"172.21.15.124:6817","nonce":90885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":90885},{"type":"v1","addr":"172.21.15.124:6819","nonce":90885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":90885},{"type":"v1","addr":"172.21.15.124:6823","nonce":90885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":90885},{"type":"v1","addr":"172.21.15.124:6821","nonce":90885}]},"public_addr":"172.21.15.124:6817/90885","cluster_addr":"172.21.15.124:6819/90885","heartbeat_back_addr":"172.21.15.124:6823/90885","heartbeat_front_addr":"172.21.15.124:6821/90885","state":["exists","up"]},{"osd":6,"uuid":"15463e53-ac8d-47d1-b4f6-c0d31f29e4a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":90864},{"type":"v1","addr":"172.21.15.124:6809","nonce":90864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":90864},{"type":"v1","addr":"172.21.15.124:6811","nonce":90864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":90864},{"type":"v1","addr":"172.21.15.124:6815","nonce":90864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":90864},{"type":"v1","addr":"172.21.15.124:6813","nonce":90864}]},"public_addr":"172.21.15.124:6809/90864","cluster_addr":"172.21.15.124:6811/90864","heartbeat_back_addr":"172.21.15.124:6815/90864","heartbeat_front_addr":"172.21.15.124:6813/90864","state":["exists","up"]},{"osd":7,"uuid":"ae463a90-a9d3-498f-86fa-47a35b3fe889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":90894},{"type":"v1","addr":"172.21.15.124:6825","nonce":90894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":90894},{"type":"v1","addr":"172.21.15.124:6827","nonce":90894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":90894},{"type":"v1","addr":"172.21.15.124:6831","nonce":90894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":90894},{"type":"v1","addr":"172.21.15.124:6829","nonce":90894}]},"public_addr":"172.21.15.124:6825/90894","cluster_addr":"172.21.15.124:6827/90894","heartbeat_back_addr":"172.21.15.124:6831/90894","heartbeat_front_addr":"172.21.15.124:6829/90894","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.827704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.839951+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.032230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.050605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.987908+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.031290+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.080707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.053692+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T01:41:39.619 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.617+0000 7f0c529e7700 1 -- 172.21.15.27:0/3111177789 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0c38066f30 msgr2=0x7f0c380693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:39.619 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.617+0000 7f0c529e7700 1 --2- 172.21.15.27:0/3111177789 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0c38066f30 0x7f0c380693e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c06fbb0 tx=0x7f0c400091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:39.619 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.617+0000 7f0c529e7700 1 -- 172.21.15.27:0/3111177789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c4c0b6170 msgr2=0x7f0c4c158bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:39.620 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.617+0000 7f0c529e7700 1 --2- 172.21.15.27:0/3111177789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c4c0b6170 0x7f0c4c158bd0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f0c4400cd60 tx=0x7f0c44001b10 comp rx=0 tx=0).stop 2023-10-06T01:41:39.620 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.617+0000 7f0c529e7700 1 -- 172.21.15.27:0/3111177789 shutdown_connections 2023-10-06T01:41:39.620 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.617+0000 7f0c529e7700 1 --2- 172.21.15.27:0/3111177789 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0c38066f30 0x7f0c380693e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:39.620 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.617+0000 7f0c529e7700 1 --2- 172.21.15.27:0/3111177789 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0c4c15d850 0x7f0c4c141ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:39.620 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.617+0000 7f0c529e7700 1 --2- 172.21.15.27:0/3111177789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c4c0b6170 0x7f0c4c158bd0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:39.620 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.617+0000 7f0c529e7700 1 --2- 172.21.15.27:0/3111177789 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c4c15d480 0x7f0c4c1419a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:39.620 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.617+0000 7f0c529e7700 1 -- 172.21.15.27:0/3111177789 >> 172.21.15.27:0/3111177789 conn(0x7f0c4c0ccf80 msgr2=0x7f0c4c15bcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:39.620 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.617+0000 7f0c529e7700 1 -- 172.21.15.27:0/3111177789 shutdown_connections 2023-10-06T01:41:39.620 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.617+0000 7f0c529e7700 1 -- 172.21.15.27:0/3111177789 wait complete. 2023-10-06T01:41:39.631 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-10-06T01:41:05.685386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-10-06T01:41:06.092995+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-06T01:41:10.434257+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-10-06T01:41:11.700262+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-06T01:41:39.631 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-06T01:41:39.752 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.751+0000 7f337359e700 1 Processor -- start 2023-10-06T01:41:39.752 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.751+0000 7f337359e700 1 -- start start 2023-10-06T01:41:39.755 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.752+0000 7f337359e700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f33741431c0 0x7f337415d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:39.755 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.752+0000 7f337359e700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33740b15b0 con 0x7f3374142df0 2023-10-06T01:41:39.755 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.752+0000 7f337359e700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33740b1880 con 0x7f337415d690 2023-10-06T01:41:39.755 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.752+0000 7f337359e700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33740b2920 con 0x7f33741431c0 2023-10-06T01:41:39.755 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.752+0000 7f3371d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f33741431c0 0x7f337415d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:39.755 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.752+0000 7f337259c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3374142df0 0x7f33740b0ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41008/0 (socket says 172.21.15.27:41008) 2023-10-06T01:41:39.755 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.752+0000 7f337259c700 1 -- 172.21.15.27:0/1293473631 learned_addr learned my addr 172.21.15.27:0/1293473631 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:39.755 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.752+0000 7f3371d9b700 1 -- 172.21.15.27:0/1293473631 >> v1:172.21.15.124:6789/0 conn(0x7f337415d690 legacy=0x7f33740b3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:39.755 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.752+0000 7f3371d9b700 1 -- 172.21.15.27:0/1293473631 >> v1:172.21.15.27:6789/0 conn(0x7f3374142df0 legacy=0x7f33740b0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:39.755 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.752+0000 7f3371d9b700 1 -- 172.21.15.27:0/1293473631 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33740b1b60 con 0x7f33741431c0 2023-10-06T01:41:39.755 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.752+0000 7f337159a700 1 -- 172.21.15.27:0/1293473631 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 360852715 0 0) 0x7f33740b15b0 con 0x7f3374142df0 2023-10-06T01:41:39.756 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.753+0000 7f3371d9b700 1 --2- 172.21.15.27:0/1293473631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33741431c0 0x7f337415d150 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f335c0098e0 tx=0x7f335c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de2ea13b95ffc3bd server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:39.756 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.753+0000 7f337159a700 1 -- 172.21.15.27:0/1293473631 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f335c009e80 con 0x7f33741431c0 2023-10-06T01:41:39.756 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.753+0000 7f337159a700 1 -- 172.21.15.27:0/1293473631 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f335c00b860 con 0x7f33741431c0 2023-10-06T01:41:39.756 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.753+0000 7f337159a700 1 -- 172.21.15.27:0/1293473631 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f335c00bbe0 con 0x7f33741431c0 2023-10-06T01:41:39.756 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.753+0000 7f337359e700 1 -- 172.21.15.27:0/1293473631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33741431c0 msgr2=0x7f337415d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:39.756 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.753+0000 7f337359e700 1 --2- 172.21.15.27:0/1293473631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33741431c0 0x7f337415d150 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f335c0098e0 tx=0x7f335c0091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:39.756 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.753+0000 7f337359e700 1 -- 172.21.15.27:0/1293473631 shutdown_connections 2023-10-06T01:41:39.756 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.753+0000 7f337359e700 1 --2- 172.21.15.27:0/1293473631 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f33741431c0 0x7f337415d150 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:39.756 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.753+0000 7f337359e700 1 -- 172.21.15.27:0/1293473631 >> 172.21.15.27:0/1293473631 conn(0x7f3374067630 msgr2=0x7f3374069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:39.756 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.753+0000 7f337359e700 1 -- 172.21.15.27:0/1293473631 shutdown_connections 2023-10-06T01:41:39.756 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.753+0000 7f337359e700 1 -- 172.21.15.27:0/1293473631 wait complete. 2023-10-06T01:41:39.759 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.754+0000 7f337359e700 1 Processor -- start 2023-10-06T01:41:39.759 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.754+0000 7f337359e700 1 -- start start 2023-10-06T01:41:39.759 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.755+0000 7f337359e700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3374142df0 0x7f3374140f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:39.759 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.755+0000 7f337359e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f33741431c0 0x7f3374155660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:39.759 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.755+0000 7f337359e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f337415d690 0x7f3374155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:39.759 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.755+0000 7f337359e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33741583f0 con 0x7f337415d690 2023-10-06T01:41:39.759 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.755+0000 7f337359e700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33741414b0 con 0x7f3374142df0 2023-10-06T01:41:39.759 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.755+0000 7f337359e700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33741577e0 con 0x7f33741431c0 2023-10-06T01:41:39.759 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.755+0000 7f3372d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f337415d690 0x7f3374155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:39.759 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.755+0000 7f3372d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f337415d690 0x7f3374155ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34254/0 (socket says 172.21.15.27:34254) 2023-10-06T01:41:39.760 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.755+0000 7f3372d9d700 1 -- 172.21.15.27:0/266416848 learned_addr learned my addr 172.21.15.27:0/266416848 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:39.760 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.755+0000 7f337259c700 1 --2- 172.21.15.27:0/266416848 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3374142df0 0x7f3374140f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:39.760 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.755+0000 7f3371d9b700 1 --2- 172.21.15.27:0/266416848 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f33741431c0 0x7f3374155660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:39.760 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.755+0000 7f3372d9d700 1 -- 172.21.15.27:0/266416848 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f33741431c0 msgr2=0x7f3374155660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:39.760 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.755+0000 7f3372d9d700 1 --2- 172.21.15.27:0/266416848 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f33741431c0 0x7f3374155660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:39.760 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.755+0000 7f3372d9d700 1 -- 172.21.15.27:0/266416848 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3374142df0 msgr2=0x7f3374140f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:39.760 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.755+0000 7f3372d9d700 1 --2- 172.21.15.27:0/266416848 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3374142df0 0x7f3374140f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:39.760 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.756+0000 7f3372d9d700 1 -- 172.21.15.27:0/266416848 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f335c009420 con 0x7f337415d690 2023-10-06T01:41:39.760 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.756+0000 7f3372d9d700 1 --2- 172.21.15.27:0/266416848 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f337415d690 0x7f3374155ba0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f336400c840 tx=0x7f3364001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:39.760 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.756+0000 7f336a7fc700 1 -- 172.21.15.27:0/266416848 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3364012630 con 0x7f337415d690 2023-10-06T01:41:39.760 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.756+0000 7f336a7fc700 1 -- 172.21.15.27:0/266416848 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33640127b0 con 0x7f337415d690 2023-10-06T01:41:39.761 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.756+0000 7f337359e700 1 -- 172.21.15.27:0/266416848 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3374157ac0 con 0x7f337415d690 2023-10-06T01:41:39.761 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.756+0000 7f336a7fc700 1 -- 172.21.15.27:0/266416848 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3364012b70 con 0x7f337415d690 2023-10-06T01:41:39.761 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.756+0000 7f337359e700 1 -- 172.21.15.27:0/266416848 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3374158080 con 0x7f337415d690 2023-10-06T01:41:39.761 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.757+0000 7f337359e700 1 -- 172.21.15.27:0/266416848 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3374158560 con 0x7f337415d690 2023-10-06T01:41:39.761 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.757+0000 7f336a7fc700 1 -- 172.21.15.27:0/266416848 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f336401a3a0 con 0x7f337415d690 2023-10-06T01:41:39.761 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.758+0000 7f336a7fc700 1 --2- 172.21.15.27:0/266416848 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3354066f30 0x7f33540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:39.761 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.758+0000 7f336a7fc700 1 -- 172.21.15.27:0/266416848 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f33640847c0 con 0x7f337415d690 2023-10-06T01:41:39.762 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.758+0000 7f337259c700 1 --2- 172.21.15.27:0/266416848 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3354066f30 0x7f33540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:39.762 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.758+0000 7f337259c700 1 --2- 172.21.15.27:0/266416848 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3354066f30 0x7f33540693e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f336c007fd0 tx=0x7f336c007d10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:39.763 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.762+0000 7f336a7fc700 1 -- 172.21.15.27:0/266416848 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f33640518d0 con 0x7f337415d690 2023-10-06T01:41:39.953 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.952+0000 7f337359e700 1 -- 172.21.15.27:0/266416848 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f337404c3c0 con 0x7f337415d690 2023-10-06T01:41:39.954 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.952+0000 7f336a7fc700 1 -- 172.21.15.27:0/266416848 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f3364054ef0 con 0x7f337415d690 2023-10-06T01:41:39.954 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 8 2023-10-06T01:41:39.956 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.954+0000 7f337359e700 1 -- 172.21.15.27:0/266416848 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3354066f30 msgr2=0x7f33540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:39.956 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.954+0000 7f337359e700 1 --2- 172.21.15.27:0/266416848 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3354066f30 0x7f33540693e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f336c007fd0 tx=0x7f336c007d10 comp rx=0 tx=0).stop 2023-10-06T01:41:39.956 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.954+0000 7f337359e700 1 -- 172.21.15.27:0/266416848 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f337415d690 msgr2=0x7f3374155ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:39.956 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.954+0000 7f337359e700 1 --2- 172.21.15.27:0/266416848 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f337415d690 0x7f3374155ba0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f336400c840 tx=0x7f3364001670 comp rx=0 tx=0).stop 2023-10-06T01:41:39.956 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.954+0000 7f337359e700 1 -- 172.21.15.27:0/266416848 shutdown_connections 2023-10-06T01:41:39.956 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.954+0000 7f337359e700 1 --2- 172.21.15.27:0/266416848 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3354066f30 0x7f33540693e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:39.956 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.954+0000 7f337359e700 1 --2- 172.21.15.27:0/266416848 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f337415d690 0x7f3374155ba0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:39.956 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.954+0000 7f337359e700 1 --2- 172.21.15.27:0/266416848 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3374142df0 0x7f3374140f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:39.957 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.954+0000 7f337359e700 1 --2- 172.21.15.27:0/266416848 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f33741431c0 0x7f3374155660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:39.957 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.954+0000 7f337359e700 1 -- 172.21.15.27:0/266416848 >> 172.21.15.27:0/266416848 conn(0x7f3374067630 msgr2=0x7f33740b0c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:39.957 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.954+0000 7f337359e700 1 -- 172.21.15.27:0/266416848 shutdown_connections 2023-10-06T01:41:39.957 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:39.954+0000 7f337359e700 1 -- 172.21.15.27:0/266416848 wait complete. 2023-10-06T01:41:39.967 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-06T01:41:40.089 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.088+0000 7fa9e23f3700 1 Processor -- start 2023-10-06T01:41:40.090 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.088+0000 7fa9e23f3700 1 -- start start 2023-10-06T01:41:40.091 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.089+0000 7fa9e23f3700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa9dc1598d0 0x7fa9dc15d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:40.091 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.089+0000 7fa9e23f3700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fa9dc15d8e0 0x7fa9dc0cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:40.091 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.089+0000 7fa9e23f3700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9dc15dd20 con 0x7fa9dc159500 2023-10-06T01:41:40.092 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.089+0000 7fa9e23f3700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9dc0b1110 con 0x7fa9dc15d8e0 2023-10-06T01:41:40.092 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.089+0000 7fa9e23f3700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc0b13b0 con 0x7fa9dc1598d0 2023-10-06T01:41:40.092 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.089+0000 7fa9dbfff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa9dc159500 0x7fa9dc0b3fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41010/0 (socket says 172.21.15.27:41010) 2023-10-06T01:41:40.092 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.089+0000 7fa9dbfff700 1 -- 172.21.15.27:0/2830541423 learned_addr learned my addr 172.21.15.27:0/2830541423 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:40.092 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.089+0000 7fa9e0990700 1 --2- 172.21.15.27:0/2830541423 >> v2:172.21.15.124:3300/0 conn(0x7fa9dc15d8e0 0x7fa9dc0cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:40.092 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.089+0000 7fa9db7fe700 1 --2- 172.21.15.27:0/2830541423 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa9dc1598d0 0x7fa9dc15d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:40.093 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.089+0000 7fa9daffd700 1 -- 172.21.15.27:0/2830541423 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2614904187 0 0) 0x7fa9dc15dd20 con 0x7fa9dc159500 2023-10-06T01:41:40.093 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.090+0000 7fa9daffd700 1 -- 172.21.15.27:0/2830541423 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9c4003570 con 0x7fa9dc159500 2023-10-06T01:41:40.093 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.090+0000 7fa9daffd700 1 -- 172.21.15.27:0/2830541423 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2688369974 0 0) 0x7fa9c4003570 con 0x7fa9dc159500 2023-10-06T01:41:40.093 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.090+0000 7fa9db7fe700 1 -- 172.21.15.27:0/2830541423 >> v2:172.21.15.124:3300/0 conn(0x7fa9dc15d8e0 msgr2=0x7fa9dc0cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:40.093 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.090+0000 7fa9db7fe700 1 --2- 172.21.15.27:0/2830541423 >> v2:172.21.15.124:3300/0 conn(0x7fa9dc15d8e0 0x7fa9dc0cf6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:40.093 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.090+0000 7fa9db7fe700 1 -- 172.21.15.27:0/2830541423 >> v1:172.21.15.27:6789/0 conn(0x7fa9dc159500 legacy=0x7fa9dc0b3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:40.094 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.090+0000 7fa9db7fe700 1 -- 172.21.15.27:0/2830541423 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9dc0b1690 con 0x7fa9dc1598d0 2023-10-06T01:41:40.094 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.090+0000 7fa9db7fe700 1 --2- 172.21.15.27:0/2830541423 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa9dc1598d0 0x7fa9dc15d3a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa9d00098e0 tx=0x7fa9d00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23f62de15ea40e39 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:40.094 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.090+0000 7fa9daffd700 1 -- 172.21.15.27:0/2830541423 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc001470 con 0x7fa9dc1598d0 2023-10-06T01:41:40.094 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.090+0000 7fa9daffd700 1 -- 172.21.15.27:0/2830541423 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d0011420 con 0x7fa9dc1598d0 2023-10-06T01:41:40.094 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.090+0000 7fa9daffd700 1 -- 172.21.15.27:0/2830541423 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d00117c0 con 0x7fa9dc1598d0 2023-10-06T01:41:40.097 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.091+0000 7fa9e23f3700 1 -- 172.21.15.27:0/2830541423 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa9dc1598d0 msgr2=0x7fa9dc15d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:40.097 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.091+0000 7fa9e23f3700 1 --2- 172.21.15.27:0/2830541423 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa9dc1598d0 0x7fa9dc15d3a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa9d00098e0 tx=0x7fa9d00091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:40.097 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.091+0000 7fa9e23f3700 1 -- 172.21.15.27:0/2830541423 shutdown_connections 2023-10-06T01:41:40.097 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.091+0000 7fa9e23f3700 1 --2- 172.21.15.27:0/2830541423 >> v2:172.21.15.124:3300/0 conn(0x7fa9dc15d8e0 0x7fa9dc0cf6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:40.098 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.091+0000 7fa9e23f3700 1 --2- 172.21.15.27:0/2830541423 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa9dc1598d0 0x7fa9dc15d3a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:40.098 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.091+0000 7fa9e23f3700 1 -- 172.21.15.27:0/2830541423 >> 172.21.15.27:0/2830541423 conn(0x7fa9dc067630 msgr2=0x7fa9dc069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:40.098 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.091+0000 7fa9e23f3700 1 -- 172.21.15.27:0/2830541423 shutdown_connections 2023-10-06T01:41:40.098 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.091+0000 7fa9e23f3700 1 -- 172.21.15.27:0/2830541423 wait complete. 2023-10-06T01:41:40.098 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.092+0000 7fa9e23f3700 1 Processor -- start 2023-10-06T01:41:40.098 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.092+0000 7fa9e23f3700 1 -- start start 2023-10-06T01:41:40.098 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.093+0000 7fa9e23f3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9dc159500 0x7fa9dc14f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:40.098 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.093+0000 7fa9e23f3700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9dc1598d0 0x7fa9dc14b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:40.098 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.093+0000 7fa9e23f3700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa9dc15d8e0 0x7fa9dc0689d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:40.098 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.093+0000 7fa9e23f3700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc153280 con 0x7fa9dc159500 2023-10-06T01:41:40.099 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.093+0000 7fa9e23f3700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc14f0a0 con 0x7fa9dc1598d0 2023-10-06T01:41:40.099 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.093+0000 7fa9e23f3700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc14e490 con 0x7fa9dc15d8e0 2023-10-06T01:41:40.099 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.093+0000 7fa9dbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9dc159500 0x7fa9dc14f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:40.099 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.093+0000 7fa9dbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9dc159500 0x7fa9dc14f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34270/0 (socket says 172.21.15.27:34270) 2023-10-06T01:41:40.099 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.093+0000 7fa9dbfff700 1 -- 172.21.15.27:0/861338933 learned_addr learned my addr 172.21.15.27:0/861338933 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:40.099 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.093+0000 7fa9e0990700 1 --2- 172.21.15.27:0/861338933 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa9dc15d8e0 0x7fa9dc0689d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:40.099 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.093+0000 7fa9db7fe700 1 --2- 172.21.15.27:0/861338933 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9dc1598d0 0x7fa9dc14b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:40.099 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.093+0000 7fa9dbfff700 1 -- 172.21.15.27:0/861338933 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa9dc15d8e0 msgr2=0x7fa9dc0689d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:40.099 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.093+0000 7fa9dbfff700 1 --2- 172.21.15.27:0/861338933 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa9dc15d8e0 0x7fa9dc0689d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:40.099 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.093+0000 7fa9dbfff700 1 -- 172.21.15.27:0/861338933 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9dc1598d0 msgr2=0x7fa9dc14b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:40.099 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.093+0000 7fa9dbfff700 1 --2- 172.21.15.27:0/861338933 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9dc1598d0 0x7fa9dc14b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:40.099 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.093+0000 7fa9dbfff700 1 -- 172.21.15.27:0/861338933 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d0009420 con 0x7fa9dc159500 2023-10-06T01:41:40.100 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.093+0000 7fa9dbfff700 1 --2- 172.21.15.27:0/861338933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9dc159500 0x7fa9dc14f530 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc00cb90 tx=0x7fa9cc001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:40.100 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.093+0000 7fa9c3fff700 1 -- 172.21.15.27:0/861338933 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc007150 con 0x7fa9dc159500 2023-10-06T01:41:40.100 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.093+0000 7fa9c3fff700 1 -- 172.21.15.27:0/861338933 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9cc0072b0 con 0x7fa9dc159500 2023-10-06T01:41:40.100 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.093+0000 7fa9e23f3700 1 -- 172.21.15.27:0/861338933 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9dc14e770 con 0x7fa9dc159500 2023-10-06T01:41:40.100 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.093+0000 7fa9c3fff700 1 -- 172.21.15.27:0/861338933 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc007650 con 0x7fa9dc159500 2023-10-06T01:41:40.100 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.094+0000 7fa9e23f3700 1 -- 172.21.15.27:0/861338933 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9dc14ed30 con 0x7fa9dc159500 2023-10-06T01:41:40.100 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.094+0000 7fa9e23f3700 1 -- 172.21.15.27:0/861338933 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9dc064050 con 0x7fa9dc159500 2023-10-06T01:41:40.100 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.095+0000 7fa9c3fff700 1 -- 172.21.15.27:0/861338933 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa9cc004a40 con 0x7fa9dc159500 2023-10-06T01:41:40.100 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.095+0000 7fa9c3fff700 1 --2- 172.21.15.27:0/861338933 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa9c8066f30 0x7fa9c80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:40.100 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.095+0000 7fa9c3fff700 1 -- 172.21.15.27:0/861338933 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fa9cc083ec0 con 0x7fa9dc159500 2023-10-06T01:41:40.101 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.095+0000 7fa9db7fe700 1 --2- 172.21.15.27:0/861338933 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa9c8066f30 0x7fa9c80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:40.101 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.096+0000 7fa9db7fe700 1 --2- 172.21.15.27:0/861338933 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa9c8066f30 0x7fa9c80693e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa9d00030c0 tx=0x7fa9d0016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:40.101 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.100+0000 7fa9c3fff700 1 -- 172.21.15.27:0/861338933 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa9cc050fd0 con 0x7fa9dc159500 2023-10-06T01:41:40.293 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.292+0000 7fa9e23f3700 1 -- 172.21.15.27:0/861338933 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa9dc0b1970 con 0x7fa9dc159500 2023-10-06T01:41:40.293 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.292+0000 7fa9c3fff700 1 -- 172.21.15.27:0/861338933 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa9cc018020 con 0x7fa9dc159500 2023-10-06T01:41:40.294 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2023-10-06T01:41:40.295 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.294+0000 7fa9e23f3700 1 -- 172.21.15.27:0/861338933 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa9c8066f30 msgr2=0x7fa9c80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:40.296 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.294+0000 7fa9e23f3700 1 --2- 172.21.15.27:0/861338933 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa9c8066f30 0x7fa9c80693e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa9d00030c0 tx=0x7fa9d0016000 comp rx=0 tx=0).stop 2023-10-06T01:41:40.296 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.294+0000 7fa9e23f3700 1 -- 172.21.15.27:0/861338933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9dc159500 msgr2=0x7fa9dc14f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:40.296 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.294+0000 7fa9e23f3700 1 --2- 172.21.15.27:0/861338933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9dc159500 0x7fa9dc14f530 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc00cb90 tx=0x7fa9cc001a80 comp rx=0 tx=0).stop 2023-10-06T01:41:40.296 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.294+0000 7fa9e23f3700 1 -- 172.21.15.27:0/861338933 shutdown_connections 2023-10-06T01:41:40.296 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.294+0000 7fa9e23f3700 1 --2- 172.21.15.27:0/861338933 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa9c8066f30 0x7fa9c80693e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:40.296 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.294+0000 7fa9e23f3700 1 --2- 172.21.15.27:0/861338933 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa9dc15d8e0 0x7fa9dc0689d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:40.296 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.294+0000 7fa9e23f3700 1 --2- 172.21.15.27:0/861338933 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa9dc159500 0x7fa9dc14f530 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:40.296 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.294+0000 7fa9e23f3700 1 --2- 172.21.15.27:0/861338933 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa9dc1598d0 0x7fa9dc14b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:40.296 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.294+0000 7fa9e23f3700 1 -- 172.21.15.27:0/861338933 >> 172.21.15.27:0/861338933 conn(0x7fa9dc067630 msgr2=0x7fa9dc156ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:40.296 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.294+0000 7fa9e23f3700 1 -- 172.21.15.27:0/861338933 shutdown_connections 2023-10-06T01:41:40.297 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.294+0000 7fa9e23f3700 1 -- 172.21.15.27:0/861338933 wait complete. 2023-10-06T01:41:40.307 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-06T01:41:40.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.428+0000 7f3202346700 1 Processor -- start 2023-10-06T01:41:40.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.428+0000 7f3202346700 1 -- start start 2023-10-06T01:41:40.431 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.429+0000 7f3202346700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f31fc1598d0 0x7f31fc15d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:40.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.429+0000 7f3202346700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31fc15dd20 con 0x7f31fc15d8e0 2023-10-06T01:41:40.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.429+0000 7f3202346700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31fc0b1110 con 0x7f31fc159500 2023-10-06T01:41:40.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.429+0000 7f3202346700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31fc0b13b0 con 0x7f31fc1598d0 2023-10-06T01:41:40.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.429+0000 7f32008e3700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f31fc15d8e0 0x7f31fc0cf6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41026/0 (socket says 172.21.15.27:41026) 2023-10-06T01:41:40.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.429+0000 7f32008e3700 1 -- 172.21.15.27:0/3054968899 learned_addr learned my addr 172.21.15.27:0/3054968899 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:40.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.429+0000 7f31fb7fe700 1 --2- 172.21.15.27:0/3054968899 >> v2:172.21.15.124:3300/0 conn(0x7f31fc1598d0 0x7f31fc15d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:40.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.429+0000 7f31faffd700 1 -- 172.21.15.27:0/3054968899 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1330621638 0 0) 0x7f31fc15dd20 con 0x7f31fc15d8e0 2023-10-06T01:41:40.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.429+0000 7f31faffd700 1 -- 172.21.15.27:0/3054968899 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31e4003570 con 0x7f31fc15d8e0 2023-10-06T01:41:40.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.429+0000 7f31fb7fe700 1 -- 172.21.15.27:0/3054968899 >> v1:172.21.15.124:6789/0 conn(0x7f31fc159500 legacy=0x7f31fc0b3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:40.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.429+0000 7f31fb7fe700 1 -- 172.21.15.27:0/3054968899 >> v1:172.21.15.27:6789/0 conn(0x7f31fc15d8e0 legacy=0x7f31fc0cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:40.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.429+0000 7f31faffd700 1 -- 172.21.15.27:0/3054968899 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 911914874 0 0) 0x7f31fc0b1110 con 0x7f31fc159500 2023-10-06T01:41:40.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.429+0000 7f31fb7fe700 1 -- 172.21.15.27:0/3054968899 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31fc0b1690 con 0x7f31fc1598d0 2023-10-06T01:41:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.430+0000 7f31fb7fe700 1 --2- 172.21.15.27:0/3054968899 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f31fc1598d0 0x7f31fc15d3a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f31f00098e0 tx=0x7f31f00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f1f828b1eec10f server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.430+0000 7f31faffd700 1 -- 172.21.15.27:0/3054968899 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f4001770 con 0x7f31fc1598d0 2023-10-06T01:41:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.430+0000 7f31faffd700 1 -- 172.21.15.27:0/3054968899 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31f0011420 con 0x7f31fc1598d0 2023-10-06T01:41:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.430+0000 7f31faffd700 1 -- 172.21.15.27:0/3054968899 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31f00117c0 con 0x7f31fc1598d0 2023-10-06T01:41:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.430+0000 7f3202346700 1 -- 172.21.15.27:0/3054968899 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f31fc1598d0 msgr2=0x7f31fc15d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.430+0000 7f3202346700 1 --2- 172.21.15.27:0/3054968899 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f31fc1598d0 0x7f31fc15d3a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f31f00098e0 tx=0x7f31f00091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.430+0000 7f3202346700 1 -- 172.21.15.27:0/3054968899 shutdown_connections 2023-10-06T01:41:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.430+0000 7f3202346700 1 --2- 172.21.15.27:0/3054968899 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f31fc1598d0 0x7f31fc15d3a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.430+0000 7f3202346700 1 -- 172.21.15.27:0/3054968899 >> 172.21.15.27:0/3054968899 conn(0x7f31fc067630 msgr2=0x7f31fc069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.430+0000 7f3202346700 1 -- 172.21.15.27:0/3054968899 shutdown_connections 2023-10-06T01:41:40.433 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.430+0000 7f3202346700 1 -- 172.21.15.27:0/3054968899 wait complete. 2023-10-06T01:41:40.435 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.431+0000 7f3202346700 1 Processor -- start 2023-10-06T01:41:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.432+0000 7f3202346700 1 -- start start 2023-10-06T01:41:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.432+0000 7f3202346700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31fc159500 0x7f31fc140330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.432+0000 7f3202346700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f31fc1598d0 0x7f31fc15ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.432+0000 7f3202346700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f31fc15d8e0 0x7f31fc15c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.432+0000 7f3202346700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31fc14cff0 con 0x7f31fc159500 2023-10-06T01:41:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.432+0000 7f3202346700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31fc15b670 con 0x7f31fc1598d0 2023-10-06T01:41:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.432+0000 7f3202346700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31fc15b950 con 0x7f31fc15d8e0 2023-10-06T01:41:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.432+0000 7f31fbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31fc159500 0x7f31fc140330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.432+0000 7f31fbfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31fc159500 0x7f31fc140330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34280/0 (socket says 172.21.15.27:34280) 2023-10-06T01:41:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.432+0000 7f31fbfff700 1 -- 172.21.15.27:0/3583393650 learned_addr learned my addr 172.21.15.27:0/3583393650 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.433+0000 7f32008e3700 1 --2- 172.21.15.27:0/3583393650 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f31fc15d8e0 0x7f31fc15c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:40.436 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.433+0000 7f31fb7fe700 1 --2- 172.21.15.27:0/3583393650 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f31fc1598d0 0x7f31fc15ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:40.437 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.433+0000 7f31fbfff700 1 -- 172.21.15.27:0/3583393650 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f31fc15d8e0 msgr2=0x7f31fc15c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:40.437 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.433+0000 7f31fbfff700 1 --2- 172.21.15.27:0/3583393650 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f31fc15d8e0 0x7f31fc15c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:40.437 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.433+0000 7f31fbfff700 1 -- 172.21.15.27:0/3583393650 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f31fc1598d0 msgr2=0x7f31fc15ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:40.437 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.433+0000 7f31fbfff700 1 --2- 172.21.15.27:0/3583393650 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f31fc1598d0 0x7f31fc15ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:40.437 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.433+0000 7f31fbfff700 1 -- 172.21.15.27:0/3583393650 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31f0009420 con 0x7f31fc159500 2023-10-06T01:41:40.437 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.433+0000 7f31fbfff700 1 --2- 172.21.15.27:0/3583393650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31fc159500 0x7f31fc140330 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f31ec00c7e0 tx=0x7f31ec007e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:40.437 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.433+0000 7f31e3fff700 1 -- 172.21.15.27:0/3583393650 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ec012630 con 0x7f31fc159500 2023-10-06T01:41:40.437 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.433+0000 7f31e3fff700 1 -- 172.21.15.27:0/3583393650 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31ec0127b0 con 0x7f31fc159500 2023-10-06T01:41:40.437 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.433+0000 7f31e3fff700 1 -- 172.21.15.27:0/3583393650 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ec012b70 con 0x7f31fc159500 2023-10-06T01:41:40.437 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.433+0000 7f3202346700 1 -- 172.21.15.27:0/3583393650 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31fc14cc80 con 0x7f31fc159500 2023-10-06T01:41:40.437 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.433+0000 7f3202346700 1 -- 172.21.15.27:0/3583393650 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31fc13d9d0 con 0x7f31fc159500 2023-10-06T01:41:40.437 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.434+0000 7f3202346700 1 -- 172.21.15.27:0/3583393650 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31fc14cae0 con 0x7f31fc159500 2023-10-06T01:41:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.434+0000 7f31e3fff700 1 -- 172.21.15.27:0/3583393650 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f31ec01b400 con 0x7f31fc159500 2023-10-06T01:41:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.435+0000 7f31e3fff700 1 --2- 172.21.15.27:0/3583393650 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f31e8066f30 0x7f31e80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.435+0000 7f31e3fff700 1 -- 172.21.15.27:0/3583393650 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f31ec084350 con 0x7f31fc159500 2023-10-06T01:41:40.438 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.435+0000 7f31fb7fe700 1 --2- 172.21.15.27:0/3583393650 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f31e8066f30 0x7f31e80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:40.439 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.435+0000 7f31fb7fe700 1 --2- 172.21.15.27:0/3583393650 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f31e8066f30 0x7f31e80693e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f31f00030c0 tx=0x7f31f0017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:40.440 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.439+0000 7f31e3fff700 1 -- 172.21.15.27:0/3583393650 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f31fc14cae0 con 0x7f31fc159500 2023-10-06T01:41:40.633 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.632+0000 7f3202346700 1 -- 172.21.15.27:0/3583393650 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f31fc14cae0 con 0x7f31fc159500 2023-10-06T01:41:40.633 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.632+0000 7f31e3fff700 1 -- 172.21.15.27:0/3583393650 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f31fc14cae0 con 0x7f31fc159500 2023-10-06T01:41:40.633 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2023-10-06T01:41:40.636 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.634+0000 7f3202346700 1 -- 172.21.15.27:0/3583393650 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f31e8066f30 msgr2=0x7f31e80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:40.636 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.634+0000 7f3202346700 1 --2- 172.21.15.27:0/3583393650 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f31e8066f30 0x7f31e80693e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f31f00030c0 tx=0x7f31f0017000 comp rx=0 tx=0).stop 2023-10-06T01:41:40.636 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.634+0000 7f3202346700 1 -- 172.21.15.27:0/3583393650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31fc159500 msgr2=0x7f31fc140330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:40.636 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.634+0000 7f3202346700 1 --2- 172.21.15.27:0/3583393650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31fc159500 0x7f31fc140330 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f31ec00c7e0 tx=0x7f31ec007e80 comp rx=0 tx=0).stop 2023-10-06T01:41:40.636 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.634+0000 7f3202346700 1 -- 172.21.15.27:0/3583393650 shutdown_connections 2023-10-06T01:41:40.636 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.634+0000 7f3202346700 1 --2- 172.21.15.27:0/3583393650 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f31e8066f30 0x7f31e80693e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:40.636 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.634+0000 7f3202346700 1 --2- 172.21.15.27:0/3583393650 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f31fc15d8e0 0x7f31fc15c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:40.636 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.634+0000 7f3202346700 1 --2- 172.21.15.27:0/3583393650 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f31fc159500 0x7f31fc140330 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:40.636 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.634+0000 7f3202346700 1 --2- 172.21.15.27:0/3583393650 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f31fc1598d0 0x7f31fc15ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:40.636 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.634+0000 7f3202346700 1 -- 172.21.15.27:0/3583393650 >> 172.21.15.27:0/3583393650 conn(0x7f31fc067630 msgr2=0x7f31fc14c360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:40.636 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.634+0000 7f3202346700 1 -- 172.21.15.27:0/3583393650 shutdown_connections 2023-10-06T01:41:40.637 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.634+0000 7f3202346700 1 -- 172.21.15.27:0/3583393650 wait complete. 2023-10-06T01:41:40.647 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-06T01:41:40.768 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.767+0000 7f6c292e1700 1 Processor -- start 2023-10-06T01:41:40.769 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.767+0000 7f6c292e1700 1 -- start start 2023-10-06T01:41:40.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.768+0000 7f6c292e1700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f6c2415de80 0x7f6c240b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:40.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.768+0000 7f6c292e1700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c24069380 con 0x7f6c24142e90 2023-10-06T01:41:40.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.768+0000 7f6c292e1700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c24069f20 con 0x7f6c240b61d0 2023-10-06T01:41:40.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.768+0000 7f6c292e1700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c24068770 con 0x7f6c2415de80 2023-10-06T01:41:40.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.768+0000 7f6c23fff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6c24142e90 0x7f6c24069810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41036/0 (socket says 172.21.15.27:41036) 2023-10-06T01:41:40.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.768+0000 7f6c23fff700 1 -- 172.21.15.27:0/2400925619 learned_addr learned my addr 172.21.15.27:0/2400925619 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:40.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.768+0000 7f6c237fe700 1 --2- 172.21.15.27:0/2400925619 >> v2:172.21.15.124:3300/0 conn(0x7f6c2415de80 0x7f6c240b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:40.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.768+0000 7f6c22ffd700 1 -- 172.21.15.27:0/2400925619 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3185391497 0 0) 0x7f6c24069380 con 0x7f6c24142e90 2023-10-06T01:41:40.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.768+0000 7f6c22ffd700 1 -- 172.21.15.27:0/2400925619 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c0c003570 con 0x7f6c24142e90 2023-10-06T01:41:40.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.769+0000 7f6c22ffd700 1 -- 172.21.15.27:0/2400925619 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1867697147 0 0) 0x7f6c0c003570 con 0x7f6c24142e90 2023-10-06T01:41:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.769+0000 7f6c22ffd700 1 -- 172.21.15.27:0/2400925619 >> v2:172.21.15.124:3300/0 conn(0x7f6c2415de80 msgr2=0x7f6c240b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.769+0000 7f6c22ffd700 1 --2- 172.21.15.27:0/2400925619 >> v2:172.21.15.124:3300/0 conn(0x7f6c2415de80 0x7f6c240b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.769+0000 7f6c22ffd700 1 -- 172.21.15.27:0/2400925619 >> v1:172.21.15.124:6789/0 conn(0x7f6c240b61d0 legacy=0x7f6c2415d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.769+0000 7f6c22ffd700 1 -- 172.21.15.27:0/2400925619 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c24068a50 con 0x7f6c24142e90 2023-10-06T01:41:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.769+0000 7f6c22ffd700 1 -- 172.21.15.27:0/2400925619 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2619078944 0 0) 0x7f6c14001470 con 0x7f6c24142e90 2023-10-06T01:41:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.769+0000 7f6c22ffd700 1 -- 172.21.15.27:0/2400925619 >> v1:172.21.15.27:6789/0 conn(0x7f6c24142e90 legacy=0x7f6c24069810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.769+0000 7f6c22ffd700 1 --2- 172.21.15.27:0/2400925619 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6c0c003a60 0x7f6c0c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.769+0000 7f6c22ffd700 1 --2- 172.21.15.27:0/2400925619 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6c0c004810 0x7f6c0c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.769+0000 7f6c22ffd700 1 --2- 172.21.15.27:0/2400925619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c0c0051c0 0x7f6c0c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.769+0000 7f6c22ffd700 1 -- 172.21.15.27:0/2400925619 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c24068770 con 0x7f6c0c0051c0 2023-10-06T01:41:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.769+0000 7f6c22ffd700 1 -- 172.21.15.27:0/2400925619 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c00c160 con 0x7f6c0c003a60 2023-10-06T01:41:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.769+0000 7f6c22ffd700 1 -- 172.21.15.27:0/2400925619 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c00c320 con 0x7f6c0c004810 2023-10-06T01:41:40.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.769+0000 7f6c28ae0700 1 --2- 172.21.15.27:0/2400925619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c0c0051c0 0x7f6c0c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:40.774 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.769+0000 7f6c23fff700 1 --2- 172.21.15.27:0/2400925619 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6c0c003a60 0x7f6c0c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:40.774 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.769+0000 7f6c237fe700 1 --2- 172.21.15.27:0/2400925619 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6c0c004810 0x7f6c0c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:40.774 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.769+0000 7f6c28ae0700 1 -- 172.21.15.27:0/2400925619 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6c0c004810 msgr2=0x7f6c0c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:40.774 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.769+0000 7f6c28ae0700 1 --2- 172.21.15.27:0/2400925619 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6c0c004810 0x7f6c0c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:40.774 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.769+0000 7f6c28ae0700 1 -- 172.21.15.27:0/2400925619 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6c0c003a60 msgr2=0x7f6c0c003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:40.774 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.769+0000 7f6c28ae0700 1 --2- 172.21.15.27:0/2400925619 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6c0c003a60 0x7f6c0c003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:40.774 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.769+0000 7f6c28ae0700 1 -- 172.21.15.27:0/2400925619 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c0c003570 con 0x7f6c0c0051c0 2023-10-06T01:41:40.774 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.769+0000 7f6c28ae0700 1 -- 172.21.15.27:0/2400925619 reap_dead start 2023-10-06T01:41:40.774 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.769+0000 7f6c28ae0700 1 --2- 172.21.15.27:0/2400925619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c0c0051c0 0x7f6c0c00ba60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c001fc0 tx=0x7f6c1c001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b35f908fc0c025a server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:40.774 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.769+0000 7f6c22ffd700 1 -- 172.21.15.27:0/2400925619 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c1c001950 con 0x7f6c0c0051c0 2023-10-06T01:41:40.774 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.769+0000 7f6c22ffd700 1 -- 172.21.15.27:0/2400925619 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c1c00c8b0 con 0x7f6c0c0051c0 2023-10-06T01:41:40.774 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.769+0000 7f6c22ffd700 1 -- 172.21.15.27:0/2400925619 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c1c00cc90 con 0x7f6c0c0051c0 2023-10-06T01:41:40.775 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.770+0000 7f6c292e1700 1 -- 172.21.15.27:0/2400925619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c0c0051c0 msgr2=0x7f6c0c00ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:40.775 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.770+0000 7f6c292e1700 1 --2- 172.21.15.27:0/2400925619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c0c0051c0 0x7f6c0c00ba60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c001fc0 tx=0x7f6c1c001640 comp rx=0 tx=0).stop 2023-10-06T01:41:40.775 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.770+0000 7f6c292e1700 1 -- 172.21.15.27:0/2400925619 shutdown_connections 2023-10-06T01:41:40.775 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.770+0000 7f6c292e1700 1 --2- 172.21.15.27:0/2400925619 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c0c0051c0 0x7f6c0c00ba60 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:40.775 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.770+0000 7f6c292e1700 1 -- 172.21.15.27:0/2400925619 >> 172.21.15.27:0/2400925619 conn(0x7f6c240ccfe0 msgr2=0x7f6c240cd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:40.775 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.770+0000 7f6c292e1700 1 -- 172.21.15.27:0/2400925619 shutdown_connections 2023-10-06T01:41:40.775 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.770+0000 7f6c292e1700 1 -- 172.21.15.27:0/2400925619 wait complete. 2023-10-06T01:41:40.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.771+0000 7f6c292e1700 1 Processor -- start 2023-10-06T01:41:40.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.771+0000 7f6c292e1700 1 -- start start 2023-10-06T01:41:40.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.772+0000 7f6c292e1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c0c0051c0 0x7f6c2416d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:40.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.772+0000 7f6c292e1700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6c2413ff70 0x7f6c2413e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:40.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.772+0000 7f6c292e1700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6c240b0370 0x7f6c240b07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:40.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.772+0000 7f6c292e1700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c240b0d20 con 0x7f6c0c0051c0 2023-10-06T01:41:40.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.772+0000 7f6c292e1700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c240c5680 con 0x7f6c240b0370 2023-10-06T01:41:40.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.772+0000 7f6c292e1700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2416daf0 con 0x7f6c2413ff70 2023-10-06T01:41:40.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.772+0000 7f6c23fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c0c0051c0 0x7f6c2416d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:40.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.772+0000 7f6c23fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c0c0051c0 0x7f6c2416d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34286/0 (socket says 172.21.15.27:34286) 2023-10-06T01:41:40.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.772+0000 7f6c23fff700 1 -- 172.21.15.27:0/3434139994 learned_addr learned my addr 172.21.15.27:0/3434139994 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:40.779 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.772+0000 7f6c237fe700 1 --2- 172.21.15.27:0/3434139994 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6c2413ff70 0x7f6c2413e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:40.779 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.772+0000 7f6c28ae0700 1 --2- 172.21.15.27:0/3434139994 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6c240b0370 0x7f6c240b07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:40.779 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.772+0000 7f6c23fff700 1 -- 172.21.15.27:0/3434139994 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6c2413ff70 msgr2=0x7f6c2413e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:40.779 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.772+0000 7f6c23fff700 1 --2- 172.21.15.27:0/3434139994 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6c2413ff70 0x7f6c2413e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:40.779 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.772+0000 7f6c23fff700 1 -- 172.21.15.27:0/3434139994 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6c240b0370 msgr2=0x7f6c240b07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:40.779 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.772+0000 7f6c23fff700 1 --2- 172.21.15.27:0/3434139994 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6c240b0370 0x7f6c240b07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:40.779 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.772+0000 7f6c23fff700 1 -- 172.21.15.27:0/3434139994 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c2416ddd0 con 0x7f6c0c0051c0 2023-10-06T01:41:40.779 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.772+0000 7f6c237fe700 1 --2- 172.21.15.27:0/3434139994 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6c2413ff70 0x7f6c2413e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:40.779 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.773+0000 7f6c23fff700 1 --2- 172.21.15.27:0/3434139994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c0c0051c0 0x7f6c2416d2a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6c14007390 tx=0x7f6c14001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:40.779 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.773+0000 7f6c0bfff700 1 -- 172.21.15.27:0/3434139994 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c14018030 con 0x7f6c0c0051c0 2023-10-06T01:41:40.779 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.773+0000 7f6c0bfff700 1 -- 172.21.15.27:0/3434139994 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c1400f6c0 con 0x7f6c0c0051c0 2023-10-06T01:41:40.779 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.773+0000 7f6c292e1700 1 -- 172.21.15.27:0/3434139994 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c2416e0b0 con 0x7f6c0c0051c0 2023-10-06T01:41:40.780 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.773+0000 7f6c0bfff700 1 -- 172.21.15.27:0/3434139994 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c1400fa60 con 0x7f6c0c0051c0 2023-10-06T01:41:40.780 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.773+0000 7f6c292e1700 1 -- 172.21.15.27:0/3434139994 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c2416d7e0 con 0x7f6c0c0051c0 2023-10-06T01:41:40.780 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.774+0000 7f6c292e1700 1 -- 172.21.15.27:0/3434139994 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c2404c3c0 con 0x7f6c0c0051c0 2023-10-06T01:41:40.780 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.774+0000 7f6c0bfff700 1 -- 172.21.15.27:0/3434139994 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6c1401b070 con 0x7f6c0c0051c0 2023-10-06T01:41:40.780 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.775+0000 7f6c0bfff700 1 --2- 172.21.15.27:0/3434139994 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6c10066f30 0x7f6c100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:40.780 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.775+0000 7f6c0bfff700 1 -- 172.21.15.27:0/3434139994 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f6c14019070 con 0x7f6c0c0051c0 2023-10-06T01:41:40.780 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.775+0000 7f6c237fe700 1 --2- 172.21.15.27:0/3434139994 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6c10066f30 0x7f6c100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:40.780 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.775+0000 7f6c237fe700 1 --2- 172.21.15.27:0/3434139994 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6c10066f30 0x7f6c100693e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6c2415db70 tx=0x7f6c18008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:40.780 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.779+0000 7f6c0bfff700 1 -- 172.21.15.27:0/3434139994 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6c14051570 con 0x7f6c0c0051c0 2023-10-06T01:41:40.971 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.970+0000 7f6c292e1700 1 -- 172.21.15.27:0/3434139994 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6c24068d30 con 0x7f6c0c0051c0 2023-10-06T01:41:40.971 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.970+0000 7f6c0bfff700 1 -- 172.21.15.27:0/3434139994 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6c14054b90 con 0x7f6c0c0051c0 2023-10-06T01:41:40.972 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2023-10-06T01:41:40.973 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.972+0000 7f6c292e1700 1 -- 172.21.15.27:0/3434139994 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6c10066f30 msgr2=0x7f6c100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:40.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.972+0000 7f6c292e1700 1 --2- 172.21.15.27:0/3434139994 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6c10066f30 0x7f6c100693e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6c2415db70 tx=0x7f6c18008040 comp rx=0 tx=0).stop 2023-10-06T01:41:40.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.972+0000 7f6c292e1700 1 -- 172.21.15.27:0/3434139994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c0c0051c0 msgr2=0x7f6c2416d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:40.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.972+0000 7f6c292e1700 1 --2- 172.21.15.27:0/3434139994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c0c0051c0 0x7f6c2416d2a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6c14007390 tx=0x7f6c14001e50 comp rx=0 tx=0).stop 2023-10-06T01:41:40.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.972+0000 7f6c292e1700 1 -- 172.21.15.27:0/3434139994 shutdown_connections 2023-10-06T01:41:40.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.972+0000 7f6c292e1700 1 --2- 172.21.15.27:0/3434139994 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6c10066f30 0x7f6c100693e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:40.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.972+0000 7f6c292e1700 1 --2- 172.21.15.27:0/3434139994 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6c240b0370 0x7f6c240b07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:40.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.972+0000 7f6c292e1700 1 --2- 172.21.15.27:0/3434139994 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6c0c0051c0 0x7f6c2416d2a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:40.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.972+0000 7f6c292e1700 1 --2- 172.21.15.27:0/3434139994 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6c2413ff70 0x7f6c2413e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:40.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.972+0000 7f6c292e1700 1 -- 172.21.15.27:0/3434139994 >> 172.21.15.27:0/3434139994 conn(0x7f6c240ccfe0 msgr2=0x7f6c240b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:40.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.972+0000 7f6c292e1700 1 -- 172.21.15.27:0/3434139994 shutdown_connections 2023-10-06T01:41:40.975 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:40.972+0000 7f6c292e1700 1 -- 172.21.15.27:0/3434139994 wait complete. 2023-10-06T01:41:40.986 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T01:41:41.104 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.103+0000 7fbb258cb700 1 Processor -- start 2023-10-06T01:41:41.104 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.103+0000 7fbb258cb700 1 -- start start 2023-10-06T01:41:41.105 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.104+0000 7fbb258cb700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbb2015d560 0x7fbb200d0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:41.105 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.104+0000 7fbb258cb700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fbb201591b0 0x7fbb20159630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:41.105 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.104+0000 7fbb258cb700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb200b2ad0 con 0x7fbb2015d930 2023-10-06T01:41:41.106 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.104+0000 7fbb258cb700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb200b4160 con 0x7fbb201591b0 2023-10-06T01:41:41.106 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.104+0000 7fbb258cb700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb20159b70 con 0x7fbb2015d560 2023-10-06T01:41:41.106 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.104+0000 7fbb1f7fe700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fbb201591b0 0x7fbb20159630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:41.106 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.104+0000 7fbb1effd700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbb2015d560 0x7fbb200d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:41.106 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.104+0000 7fbb1effd700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbb2015d560 0x7fbb200d0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.27:42552/0 (socket says 172.21.15.27:42552) 2023-10-06T01:41:41.106 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.104+0000 7fbb1effd700 1 -- 172.21.15.27:0/848503578 learned_addr learned my addr 172.21.15.27:0/848503578 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:41.107 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.105+0000 7fbb1effd700 1 -- 172.21.15.27:0/848503578 >> v2:172.21.15.124:3300/0 conn(0x7fbb201591b0 msgr2=0x7fbb20159630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:41.108 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.105+0000 7fbb1effd700 1 --2- 172.21.15.27:0/848503578 >> v2:172.21.15.124:3300/0 conn(0x7fbb201591b0 0x7fbb20159630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:41.108 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.105+0000 7fbb1effd700 1 -- 172.21.15.27:0/848503578 >> v1:172.21.15.124:6789/0 conn(0x7fbb2015d930 legacy=0x7fbb200b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:41.108 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.105+0000 7fbb1dffb700 1 -- 172.21.15.27:0/848503578 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 652882810 0 0) 0x7fbb200b2ad0 con 0x7fbb2015d930 2023-10-06T01:41:41.108 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.105+0000 7fbb1effd700 1 -- 172.21.15.27:0/848503578 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb200b2230 con 0x7fbb2015d560 2023-10-06T01:41:41.108 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.105+0000 7fbb1f7fe700 1 --2- 172.21.15.27:0/848503578 >> v2:172.21.15.124:3300/0 conn(0x7fbb201591b0 0x7fbb20159630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:41:41.108 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.105+0000 7fbb1effd700 1 --2- 172.21.15.27:0/848503578 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbb2015d560 0x7fbb200d0030 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbb10009f20 tx=0x7fbb10009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7754261a8dd5dfc server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:41.108 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.105+0000 7fbb1dffb700 1 -- 172.21.15.27:0/848503578 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1000e070 con 0x7fbb2015d560 2023-10-06T01:41:41.108 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.105+0000 7fbb1dffb700 1 -- 172.21.15.27:0/848503578 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb1000b9d0 con 0x7fbb2015d560 2023-10-06T01:41:41.108 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.105+0000 7fbb1dffb700 1 -- 172.21.15.27:0/848503578 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb100092e0 con 0x7fbb2015d560 2023-10-06T01:41:41.108 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.105+0000 7fbb258cb700 1 -- 172.21.15.27:0/848503578 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbb2015d560 msgr2=0x7fbb200d0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:41.108 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.105+0000 7fbb258cb700 1 --2- 172.21.15.27:0/848503578 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbb2015d560 0x7fbb200d0030 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbb10009f20 tx=0x7fbb10009840 comp rx=0 tx=0).stop 2023-10-06T01:41:41.109 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.106+0000 7fbb258cb700 1 -- 172.21.15.27:0/848503578 shutdown_connections 2023-10-06T01:41:41.109 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.106+0000 7fbb258cb700 1 --2- 172.21.15.27:0/848503578 >> v2:172.21.15.124:3300/0 conn(0x7fbb201591b0 0x7fbb20159630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:41.109 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.106+0000 7fbb258cb700 1 --2- 172.21.15.27:0/848503578 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbb2015d560 0x7fbb200d0030 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:41.109 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.106+0000 7fbb258cb700 1 -- 172.21.15.27:0/848503578 >> 172.21.15.27:0/848503578 conn(0x7fbb200d1180 msgr2=0x7fbb20069640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:41.109 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.106+0000 7fbb258cb700 1 -- 172.21.15.27:0/848503578 shutdown_connections 2023-10-06T01:41:41.109 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.106+0000 7fbb258cb700 1 -- 172.21.15.27:0/848503578 wait complete. 2023-10-06T01:41:41.111 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.107+0000 7fbb258cb700 1 Processor -- start 2023-10-06T01:41:41.111 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.107+0000 7fbb258cb700 1 -- start start 2023-10-06T01:41:41.111 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.108+0000 7fbb258cb700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbb201591b0 0x7fbb20159d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:41.112 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.108+0000 7fbb258cb700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbb2015d560 0x7fbb20067630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:41.112 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.108+0000 7fbb258cb700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb2015d930 0x7fbb20067b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:41.112 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.108+0000 7fbb258cb700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb20151c00 con 0x7fbb2015d930 2023-10-06T01:41:41.112 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.108+0000 7fbb258cb700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb2015d050 con 0x7fbb2015d560 2023-10-06T01:41:41.112 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.108+0000 7fbb258cb700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb20150ff0 con 0x7fbb201591b0 2023-10-06T01:41:41.112 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.108+0000 7fbb1f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb2015d930 0x7fbb20067b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:41.112 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.108+0000 7fbb1effd700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbb201591b0 0x7fbb20159d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:41.112 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.108+0000 7fbb1f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb2015d930 0x7fbb20067b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34294/0 (socket says 172.21.15.27:34294) 2023-10-06T01:41:41.112 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.108+0000 7fbb1f7fe700 1 -- 172.21.15.27:0/3517321835 learned_addr learned my addr 172.21.15.27:0/3517321835 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:41.112 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.108+0000 7fbb1e7fc700 1 --2- 172.21.15.27:0/3517321835 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbb2015d560 0x7fbb20067630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:41.112 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.108+0000 7fbb1f7fe700 1 -- 172.21.15.27:0/3517321835 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbb201591b0 msgr2=0x7fbb20159d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:41.113 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.108+0000 7fbb1f7fe700 1 --2- 172.21.15.27:0/3517321835 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbb201591b0 0x7fbb20159d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:41.113 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.108+0000 7fbb1f7fe700 1 -- 172.21.15.27:0/3517321835 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbb2015d560 msgr2=0x7fbb20067630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:41.113 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.108+0000 7fbb1f7fe700 1 --2- 172.21.15.27:0/3517321835 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbb2015d560 0x7fbb20067630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:41.113 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.108+0000 7fbb1f7fe700 1 -- 172.21.15.27:0/3517321835 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb18009420 con 0x7fbb2015d930 2023-10-06T01:41:41.113 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.108+0000 7fbb1f7fe700 1 --2- 172.21.15.27:0/3517321835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb2015d930 0x7fbb20067b70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbb1800ec80 tx=0x7fbb180091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:41.113 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.108+0000 7fbb16ffd700 1 -- 172.21.15.27:0/3517321835 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1800c850 con 0x7fbb2015d930 2023-10-06T01:41:41.113 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.108+0000 7fbb16ffd700 1 -- 172.21.15.27:0/3517321835 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb1800c9b0 con 0x7fbb2015d930 2023-10-06T01:41:41.113 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.108+0000 7fbb258cb700 1 -- 172.21.15.27:0/3517321835 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb10009ae0 con 0x7fbb2015d930 2023-10-06T01:41:41.113 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.108+0000 7fbb16ffd700 1 -- 172.21.15.27:0/3517321835 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1800cd50 con 0x7fbb2015d930 2023-10-06T01:41:41.113 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.108+0000 7fbb258cb700 1 -- 172.21.15.27:0/3517321835 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb201515b0 con 0x7fbb2015d930 2023-10-06T01:41:41.113 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.109+0000 7fbb258cb700 1 -- 172.21.15.27:0/3517321835 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb20152780 con 0x7fbb2015d930 2023-10-06T01:41:41.114 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.110+0000 7fbb16ffd700 1 -- 172.21.15.27:0/3517321835 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fbb18013680 con 0x7fbb2015d930 2023-10-06T01:41:41.114 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.110+0000 7fbb16ffd700 1 --2- 172.21.15.27:0/3517321835 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fbb00066f30 0x7fbb000693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:41.114 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.110+0000 7fbb16ffd700 1 -- 172.21.15.27:0/3517321835 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fbb18084a40 con 0x7fbb2015d930 2023-10-06T01:41:41.114 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.110+0000 7fbb1effd700 1 --2- 172.21.15.27:0/3517321835 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fbb00066f30 0x7fbb000693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:41.114 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.111+0000 7fbb1effd700 1 --2- 172.21.15.27:0/3517321835 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fbb00066f30 0x7fbb000693e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbb10009880 tx=0x7fbb10018000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:41.115 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.114+0000 7fbb16ffd700 1 -- 172.21.15.27:0/3517321835 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbb18086e60 con 0x7fbb2015d930 2023-10-06T01:41:41.311 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.309+0000 7fbb258cb700 1 -- 172.21.15.27:0/3517321835 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb20064050 con 0x7fbb2015d930 2023-10-06T01:41:41.311 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.310+0000 7fbb16ffd700 1 -- 172.21.15.27:0/3517321835 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16866 (secure 0 0 0) 0x7fbb18051b50 con 0x7fbb2015d930 2023-10-06T01:41:41.312 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:41.312 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":20,"fsid":"20a7bff0-74f8-4f2f-b6c6-899e804bc965","created":"2023-10-06T01:40:54.474215+0000","modified":"2023-10-06T01:41:14.324546+0000","last_up_change":"2023-10-06T01:41:05.242847+0000","last_in_change":"2023-10-06T01:40:56.890960+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-10-06T01:41:05.685386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-10-06T01:41:06.092995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T01:41:10.434257+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T01:41:11.700262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b902c86d-800c-40c5-b152-b545e50d02c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":90644},{"type":"v1","addr":"172.21.15.27:6801","nonce":90644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":90644},{"type":"v1","addr":"172.21.15.27:6803","nonce":90644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":90644},{"type":"v1","addr":"172.21.15.27:6807","nonce":90644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":90644},{"type":"v1","addr":"172.21.15.27:6805","nonce":90644}]},"public_addr":"172.21.15.27:6801/90644","cluster_addr":"172.21.15.27:6803/90644","heartbeat_back_addr":"172.21.15.27:6807/90644","heartbeat_front_addr":"172.21.15.27:6805/90644","state":["exists","up"]},{"osd":1,"uuid":"5b4a70e5-b06e-48cd-bfc4-7962c5d05386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":90643},{"type":"v1","addr":"172.21.15.27:6809","nonce":90643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":90643},{"type":"v1","addr":"172.21.15.27:6811","nonce":90643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":90643},{"type":"v1","addr":"172.21.15.27:6815","nonce":90643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":90643},{"type":"v1","addr":"172.21.15.27:6813","nonce":90643}]},"public_addr":"172.21.15.27:6809/90643","cluster_addr":"172.21.15.27:6811/90643","heartbeat_back_addr":"172.21.15.27:6815/90643","heartbeat_front_addr":"172.21.15.27:6813/90643","state":["exists","up"]},{"osd":2,"uuid":"22d9de46-4073-4336-9122-5227e601eca0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":90645},{"type":"v1","addr":"172.21.15.27:6817","nonce":90645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":90645},{"type":"v1","addr":"172.21.15.27:6819","nonce":90645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":90645},{"type":"v1","addr":"172.21.15.27:6823","nonce":90645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":90645},{"type":"v1","addr":"172.21.15.27:6821","nonce":90645}]},"public_addr":"172.21.15.27:6817/90645","cluster_addr":"172.21.15.27:6819/90645","heartbeat_back_addr":"172.21.15.27:6823/90645","heartbeat_front_addr":"172.21.15.27:6821/90645","state":["exists","up"]},{"osd":3,"uuid":"438bb2d3-34e2-4b7c-9734-8360c1b67650","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":90668},{"type":"v1","addr":"172.21.15.27:6825","nonce":90668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":90668},{"type":"v1","addr":"172.21.15.27:6827","nonce":90668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":90668},{"type":"v1","addr":"172.21.15.27:6831","nonce":90668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":90668},{"type":"v1","addr":"172.21.15.27:6829","nonce":90668}]},"public_addr":"172.21.15.27:6825/90668","cluster_addr":"172.21.15.27:6827/90668","heartbeat_back_addr":"172.21.15.27:6831/90668","heartbeat_front_addr":"172.21.15.27:6829/90668","state":["exists","up"]},{"osd":4,"uuid":"e1956a8e-e0f8-4e7c-acbb-40037415a354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":90863},{"type":"v1","addr":"172.21.15.124:6801","nonce":90863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":90863},{"type":"v1","addr":"172.21.15.124:6803","nonce":90863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":90863},{"type":"v1","addr":"172.21.15.124:6807","nonce":90863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":90863},{"type":"v1","addr":"172.21.15.124:6805","nonce":90863}]},"public_addr":"172.21.15.124:6801/90863","cluster_addr":"172.21.15.124:6803/90863","heartbeat_back_addr":"172.21.15.124:6807/90863","heartbeat_front_addr":"172.21.15.124:6805/90863","state":["exists","up"]},{"osd":5,"uuid":"306c59d4-846d-4b73-b1c9-051bec9e9368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":90885},{"type":"v1","addr":"172.21.15.124:6817","nonce":90885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":90885},{"type":"v1","addr":"172.21.15.124:6819","nonce":90885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":90885},{"type":"v1","addr":"172.21.15.124:6823","nonce":90885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":90885},{"type":"v1","addr":"172.21.15.124:6821","nonce":90885}]},"public_addr":"172.21.15.124:6817/90885","cluster_addr":"172.21.15.124:6819/90885","heartbeat_back_addr":"172.21.15.124:6823/90885","heartbeat_front_addr":"172.21.15.124:6821/90885","state":["exists","up"]},{"osd":6,"uuid":"15463e53-ac8d-47d1-b4f6-c0d31f29e4a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":90864},{"type":"v1","addr":"172.21.15.124:6809","nonce":90864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":90864},{"type":"v1","addr":"172.21.15.124:6811","nonce":90864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":90864},{"type":"v1","addr":"172.21.15.124:6815","nonce":90864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":90864},{"type":"v1","addr":"172.21.15.124:6813","nonce":90864}]},"public_addr":"172.21.15.124:6809/90864","cluster_addr":"172.21.15.124:6811/90864","heartbeat_back_addr":"172.21.15.124:6815/90864","heartbeat_front_addr":"172.21.15.124:6813/90864","state":["exists","up"]},{"osd":7,"uuid":"ae463a90-a9d3-498f-86fa-47a35b3fe889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":90894},{"type":"v1","addr":"172.21.15.124:6825","nonce":90894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":90894},{"type":"v1","addr":"172.21.15.124:6827","nonce":90894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":90894},{"type":"v1","addr":"172.21.15.124:6831","nonce":90894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":90894},{"type":"v1","addr":"172.21.15.124:6829","nonce":90894}]},"public_addr":"172.21.15.124:6825/90894","cluster_addr":"172.21.15.124:6827/90894","heartbeat_back_addr":"172.21.15.124:6831/90894","heartbeat_front_addr":"172.21.15.124:6829/90894","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.827704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.839951+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.032230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.050605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.987908+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.031290+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.080707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.053692+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T01:41:41.314 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.312+0000 7fbb258cb700 1 -- 172.21.15.27:0/3517321835 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fbb00066f30 msgr2=0x7fbb000693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:41.314 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.312+0000 7fbb258cb700 1 --2- 172.21.15.27:0/3517321835 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fbb00066f30 0x7fbb000693e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fbb10009880 tx=0x7fbb10018000 comp rx=0 tx=0).stop 2023-10-06T01:41:41.314 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.312+0000 7fbb258cb700 1 -- 172.21.15.27:0/3517321835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb2015d930 msgr2=0x7fbb20067b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:41.314 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.312+0000 7fbb258cb700 1 --2- 172.21.15.27:0/3517321835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb2015d930 0x7fbb20067b70 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fbb1800ec80 tx=0x7fbb180091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:41.314 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.312+0000 7fbb258cb700 1 -- 172.21.15.27:0/3517321835 shutdown_connections 2023-10-06T01:41:41.314 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.312+0000 7fbb258cb700 1 --2- 172.21.15.27:0/3517321835 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fbb00066f30 0x7fbb000693e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:41.314 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.312+0000 7fbb258cb700 1 --2- 172.21.15.27:0/3517321835 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbb201591b0 0x7fbb20159d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:41.314 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.312+0000 7fbb258cb700 1 --2- 172.21.15.27:0/3517321835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbb2015d930 0x7fbb20067b70 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:41.315 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.312+0000 7fbb258cb700 1 --2- 172.21.15.27:0/3517321835 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbb2015d560 0x7fbb20067630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:41.315 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.312+0000 7fbb258cb700 1 -- 172.21.15.27:0/3517321835 >> 172.21.15.27:0/3517321835 conn(0x7fbb200d1180 msgr2=0x7fbb20150220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:41.315 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.312+0000 7fbb258cb700 1 -- 172.21.15.27:0/3517321835 shutdown_connections 2023-10-06T01:41:41.315 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.312+0000 7fbb258cb700 1 -- 172.21.15.27:0/3517321835 wait complete. 2023-10-06T01:41:41.326 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-10-06T01:41:05.685386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-10-06T01:41:06.092995+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-06T01:41:10.434257+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-10-06T01:41:11.700262+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-06T01:41:41.326 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-06T01:41:41.447 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.446+0000 7f7d2d086700 1 Processor -- start 2023-10-06T01:41:41.448 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.446+0000 7f7d2d086700 1 -- start start 2023-10-06T01:41:41.450 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.447+0000 7f7d2d086700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f7d28159500 0x7f7d281598d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:41.450 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.447+0000 7f7d2d086700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7d2815dab0 0x7f7d280cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:41.450 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.447+0000 7f7d2d086700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d280b1110 con 0x7f7d280b3fd0 2023-10-06T01:41:41.450 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.447+0000 7f7d2d086700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d280b13b0 con 0x7f7d28159500 2023-10-06T01:41:41.451 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.447+0000 7f7d2d086700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d280b1690 con 0x7f7d2815dab0 2023-10-06T01:41:41.451 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.447+0000 7f7d2759e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7d2815dab0 0x7f7d280cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:41.451 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.447+0000 7f7d26d9d700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f7d28159500 0x7f7d281598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:41.451 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.447+0000 7f7d2759e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7d2815dab0 0x7f7d280cf6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.27:42576/0 (socket says 172.21.15.27:42576) 2023-10-06T01:41:41.451 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.447+0000 7f7d2759e700 1 -- 172.21.15.27:0/2294485098 learned_addr learned my addr 172.21.15.27:0/2294485098 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:41.451 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.448+0000 7f7d2759e700 1 -- 172.21.15.27:0/2294485098 >> v2:172.21.15.124:3300/0 conn(0x7f7d28159500 msgr2=0x7f7d281598d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:41.451 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.448+0000 7f7d2759e700 1 --2- 172.21.15.27:0/2294485098 >> v2:172.21.15.124:3300/0 conn(0x7f7d28159500 0x7f7d281598d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:41.451 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.448+0000 7f7d2759e700 1 -- 172.21.15.27:0/2294485098 >> v1:172.21.15.124:6789/0 conn(0x7f7d280b3fd0 legacy=0x7f7d2815d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:41.451 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.448+0000 7f7d2759e700 1 -- 172.21.15.27:0/2294485098 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d280b1970 con 0x7f7d2815dab0 2023-10-06T01:41:41.452 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.448+0000 7f7d26d9d700 1 --2- 172.21.15.27:0/2294485098 >> v2:172.21.15.124:3300/0 conn(0x7f7d28159500 0x7f7d281598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:41:41.452 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.448+0000 7f7d2759e700 1 --2- 172.21.15.27:0/2294485098 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7d2815dab0 0x7f7d280cf6c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7d18009f20 tx=0x7f7d18009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24a6518c33cedf5c server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:41.452 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.448+0000 7f7d25d9b700 1 -- 172.21.15.27:0/2294485098 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d1800e070 con 0x7f7d2815dab0 2023-10-06T01:41:41.452 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.448+0000 7f7d25d9b700 1 -- 172.21.15.27:0/2294485098 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d1800b9d0 con 0x7f7d2815dab0 2023-10-06T01:41:41.452 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.448+0000 7f7d25d9b700 1 -- 172.21.15.27:0/2294485098 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d180092e0 con 0x7f7d2815dab0 2023-10-06T01:41:41.452 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.449+0000 7f7d2d086700 1 -- 172.21.15.27:0/2294485098 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7d2815dab0 msgr2=0x7f7d280cf6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:41.452 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.449+0000 7f7d2d086700 1 --2- 172.21.15.27:0/2294485098 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7d2815dab0 0x7f7d280cf6c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7d18009f20 tx=0x7f7d18009840 comp rx=0 tx=0).stop 2023-10-06T01:41:41.452 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.449+0000 7f7d2d086700 1 -- 172.21.15.27:0/2294485098 shutdown_connections 2023-10-06T01:41:41.452 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.449+0000 7f7d2d086700 1 --2- 172.21.15.27:0/2294485098 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7d2815dab0 0x7f7d280cf6c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:41.452 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.449+0000 7f7d2d086700 1 --2- 172.21.15.27:0/2294485098 >> v2:172.21.15.124:3300/0 conn(0x7f7d28159500 0x7f7d281598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:41.453 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.449+0000 7f7d2d086700 1 -- 172.21.15.27:0/2294485098 >> 172.21.15.27:0/2294485098 conn(0x7f7d28067630 msgr2=0x7f7d28069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:41.453 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.449+0000 7f7d2d086700 1 -- 172.21.15.27:0/2294485098 shutdown_connections 2023-10-06T01:41:41.453 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.449+0000 7f7d2d086700 1 -- 172.21.15.27:0/2294485098 wait complete. 2023-10-06T01:41:41.456 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.450+0000 7f7d2d086700 1 Processor -- start 2023-10-06T01:41:41.456 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.450+0000 7f7d2d086700 1 -- start start 2023-10-06T01:41:41.456 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.451+0000 7f7d2d086700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d280b3fd0 0x7f7d2813cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:41.456 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.451+0000 7f7d2d086700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7d28159500 0x7f7d2814d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:41.456 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.451+0000 7f7d2d086700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7d2815dab0 0x7f7d2814ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:41.456 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.451+0000 7f7d2d086700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d28069fc0 con 0x7f7d280b3fd0 2023-10-06T01:41:41.456 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.451+0000 7f7d2d086700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d28069720 con 0x7f7d2815dab0 2023-10-06T01:41:41.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.451+0000 7f7d2d086700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d28069a00 con 0x7f7d28159500 2023-10-06T01:41:41.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.451+0000 7f7d26d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d280b3fd0 0x7f7d2813cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:41.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.451+0000 7f7d26d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d280b3fd0 0x7f7d2813cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34302/0 (socket says 172.21.15.27:34302) 2023-10-06T01:41:41.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.451+0000 7f7d26d9d700 1 -- 172.21.15.27:0/3128357832 learned_addr learned my addr 172.21.15.27:0/3128357832 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:41.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.451+0000 7f7d2759e700 1 --2- 172.21.15.27:0/3128357832 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7d2815dab0 0x7f7d2814ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:41.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.451+0000 7f7d2659c700 1 --2- 172.21.15.27:0/3128357832 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7d28159500 0x7f7d2814d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:41.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.451+0000 7f7d26d9d700 1 -- 172.21.15.27:0/3128357832 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7d28159500 msgr2=0x7f7d2814d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:41.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.451+0000 7f7d26d9d700 1 --2- 172.21.15.27:0/3128357832 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7d28159500 0x7f7d2814d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:41.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.451+0000 7f7d26d9d700 1 -- 172.21.15.27:0/3128357832 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7d2815dab0 msgr2=0x7f7d2814ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:41.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.451+0000 7f7d26d9d700 1 --2- 172.21.15.27:0/3128357832 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7d2815dab0 0x7f7d2814ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:41.458 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.451+0000 7f7d26d9d700 1 -- 172.21.15.27:0/3128357832 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d20009420 con 0x7f7d280b3fd0 2023-10-06T01:41:41.458 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.451+0000 7f7d26d9d700 1 --2- 172.21.15.27:0/3128357832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d280b3fd0 0x7f7d2813cf60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7d2000ec80 tx=0x7f7d200091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:41.458 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.451+0000 7f7d1effd700 1 -- 172.21.15.27:0/3128357832 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2000c920 con 0x7f7d280b3fd0 2023-10-06T01:41:41.458 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.451+0000 7f7d1effd700 1 -- 172.21.15.27:0/3128357832 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d2000ca80 con 0x7f7d280b3fd0 2023-10-06T01:41:41.458 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.451+0000 7f7d2d086700 1 -- 172.21.15.27:0/3128357832 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d18009ae0 con 0x7f7d280b3fd0 2023-10-06T01:41:41.458 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.451+0000 7f7d1effd700 1 -- 172.21.15.27:0/3128357832 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2000ce00 con 0x7f7d280b3fd0 2023-10-06T01:41:41.458 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.452+0000 7f7d2d086700 1 -- 172.21.15.27:0/3128357832 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d28153ba0 con 0x7f7d280b3fd0 2023-10-06T01:41:41.458 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.452+0000 7f7d2d086700 1 -- 172.21.15.27:0/3128357832 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d28153f10 con 0x7f7d280b3fd0 2023-10-06T01:41:41.458 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.453+0000 7f7d1effd700 1 -- 172.21.15.27:0/3128357832 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7d20013680 con 0x7f7d280b3fd0 2023-10-06T01:41:41.458 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.453+0000 7f7d1effd700 1 --2- 172.21.15.27:0/3128357832 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7d08066f30 0x7f7d080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:41.459 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.453+0000 7f7d1effd700 1 -- 172.21.15.27:0/3128357832 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f7d20084db0 con 0x7f7d280b3fd0 2023-10-06T01:41:41.459 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.454+0000 7f7d2659c700 1 --2- 172.21.15.27:0/3128357832 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7d08066f30 0x7f7d080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:41.459 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.454+0000 7f7d2659c700 1 --2- 172.21.15.27:0/3128357832 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7d08066f30 0x7f7d080693e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7d281535b0 tx=0x7f7d10009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:41.459 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.458+0000 7f7d1effd700 1 -- 172.21.15.27:0/3128357832 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7d20051ec0 con 0x7f7d280b3fd0 2023-10-06T01:41:41.652 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.650+0000 7f7d2d086700 1 -- 172.21.15.27:0/3128357832 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f7d28064050 con 0x7f7d280b3fd0 2023-10-06T01:41:41.652 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.651+0000 7f7d1effd700 1 -- 172.21.15.27:0/3128357832 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f7d200554e0 con 0x7f7d280b3fd0 2023-10-06T01:41:41.652 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 8 2023-10-06T01:41:41.654 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.653+0000 7f7d2d086700 1 -- 172.21.15.27:0/3128357832 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7d08066f30 msgr2=0x7f7d080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:41.655 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.653+0000 7f7d2d086700 1 --2- 172.21.15.27:0/3128357832 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7d08066f30 0x7f7d080693e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f7d281535b0 tx=0x7f7d10009040 comp rx=0 tx=0).stop 2023-10-06T01:41:41.655 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.653+0000 7f7d2d086700 1 -- 172.21.15.27:0/3128357832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d280b3fd0 msgr2=0x7f7d2813cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:41.655 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.653+0000 7f7d2d086700 1 --2- 172.21.15.27:0/3128357832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d280b3fd0 0x7f7d2813cf60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7d2000ec80 tx=0x7f7d200091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:41.655 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.653+0000 7f7d2d086700 1 -- 172.21.15.27:0/3128357832 shutdown_connections 2023-10-06T01:41:41.655 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.653+0000 7f7d2d086700 1 --2- 172.21.15.27:0/3128357832 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7d08066f30 0x7f7d080693e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:41.655 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.653+0000 7f7d2d086700 1 --2- 172.21.15.27:0/3128357832 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7d2815dab0 0x7f7d2814ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:41.655 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.653+0000 7f7d2d086700 1 --2- 172.21.15.27:0/3128357832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7d280b3fd0 0x7f7d2813cf60 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:41.655 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.653+0000 7f7d2d086700 1 --2- 172.21.15.27:0/3128357832 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7d28159500 0x7f7d2814d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:41.655 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.653+0000 7f7d2d086700 1 -- 172.21.15.27:0/3128357832 >> 172.21.15.27:0/3128357832 conn(0x7f7d28067630 msgr2=0x7f7d280b5e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:41.655 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.653+0000 7f7d2d086700 1 -- 172.21.15.27:0/3128357832 shutdown_connections 2023-10-06T01:41:41.655 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.653+0000 7f7d2d086700 1 -- 172.21.15.27:0/3128357832 wait complete. 2023-10-06T01:41:41.666 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-06T01:41:41.787 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.786+0000 7f182ddbb700 1 Processor -- start 2023-10-06T01:41:41.787 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.786+0000 7f182ddbb700 1 -- start start 2023-10-06T01:41:41.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.787+0000 7f182ddbb700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f182815de80 0x7f18280b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:41.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.787+0000 7f182ddbb700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f18280b61d0 0x7f182815d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:41.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.787+0000 7f182ddbb700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1828069380 con 0x7f1828142e90 2023-10-06T01:41:41.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.787+0000 7f182ddbb700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1828069f20 con 0x7f182815de80 2023-10-06T01:41:41.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.787+0000 7f182ddbb700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1828068770 con 0x7f18280b61d0 2023-10-06T01:41:41.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.787+0000 7f182d5ba700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f18280b61d0 0x7f182815d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:41.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.787+0000 7f1827fff700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f182815de80 0x7f18280b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:41.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.787+0000 7f1827fff700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f182815de80 0x7f18280b5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.27:41218/0 (socket says 172.21.15.27:41218) 2023-10-06T01:41:41.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.787+0000 7f182d5ba700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f18280b61d0 0x7f182815d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.27:42600/0 (socket says 172.21.15.27:42600) 2023-10-06T01:41:41.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.787+0000 7f1827fff700 1 -- 172.21.15.27:0/3181898185 learned_addr learned my addr 172.21.15.27:0/3181898185 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:41.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.787+0000 7f182cdb9700 1 --1- 172.21.15.27:0/3181898185 >> v1:172.21.15.124:6789/0 conn(0x7f1828142e90 0x7f1828069810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.27:55980/0 (socket says 172.21.15.27:55980) 2023-10-06T01:41:41.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.787+0000 7f182d5ba700 1 -- 172.21.15.27:0/3181898185 >> v2:172.21.15.124:3300/0 conn(0x7f182815de80 msgr2=0x7f18280b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:41.798 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.787+0000 7f182d5ba700 1 --2- 172.21.15.27:0/3181898185 >> v2:172.21.15.124:3300/0 conn(0x7f182815de80 0x7f18280b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:41.798 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.787+0000 7f182d5ba700 1 -- 172.21.15.27:0/3181898185 >> v1:172.21.15.124:6789/0 conn(0x7f1828142e90 legacy=0x7f1828069810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:41.798 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.788+0000 7f182d5ba700 1 -- 172.21.15.27:0/3181898185 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1828068a50 con 0x7f18280b61d0 2023-10-06T01:41:41.798 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.788+0000 7f1827fff700 1 --2- 172.21.15.27:0/3181898185 >> v2:172.21.15.124:3300/0 conn(0x7f182815de80 0x7f18280b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:41:41.798 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.788+0000 7f182d5ba700 1 --2- 172.21.15.27:0/3181898185 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f18280b61d0 0x7f182815d290 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1820009e60 tx=0x7f1820009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99f68c6bab2244b1 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:41.798 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.788+0000 7f18277fe700 1 -- 172.21.15.27:0/3181898185 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182000e070 con 0x7f18280b61d0 2023-10-06T01:41:41.798 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.788+0000 7f18277fe700 1 -- 172.21.15.27:0/3181898185 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f182000bb60 con 0x7f18280b61d0 2023-10-06T01:41:41.798 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.788+0000 7f18277fe700 1 -- 172.21.15.27:0/3181898185 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1820012430 con 0x7f18280b61d0 2023-10-06T01:41:41.799 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.788+0000 7f182ddbb700 1 -- 172.21.15.27:0/3181898185 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f18280b61d0 msgr2=0x7f182815d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:41.799 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.788+0000 7f182ddbb700 1 --2- 172.21.15.27:0/3181898185 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f18280b61d0 0x7f182815d290 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1820009e60 tx=0x7f1820009780 comp rx=0 tx=0).stop 2023-10-06T01:41:41.799 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.788+0000 7f182ddbb700 1 -- 172.21.15.27:0/3181898185 shutdown_connections 2023-10-06T01:41:41.799 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.788+0000 7f182ddbb700 1 --2- 172.21.15.27:0/3181898185 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f18280b61d0 0x7f182815d290 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:41.799 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.788+0000 7f182ddbb700 1 --2- 172.21.15.27:0/3181898185 >> v2:172.21.15.124:3300/0 conn(0x7f182815de80 0x7f18280b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:41.799 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.788+0000 7f182ddbb700 1 -- 172.21.15.27:0/3181898185 >> 172.21.15.27:0/3181898185 conn(0x7f18280ccfe0 msgr2=0x7f18280cd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:41.799 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.788+0000 7f182ddbb700 1 -- 172.21.15.27:0/3181898185 shutdown_connections 2023-10-06T01:41:41.799 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.788+0000 7f182ddbb700 1 -- 172.21.15.27:0/3181898185 wait complete. 2023-10-06T01:41:41.799 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.789+0000 7f182ddbb700 1 Processor -- start 2023-10-06T01:41:41.799 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.790+0000 7f182ddbb700 1 -- start start 2023-10-06T01:41:41.800 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.790+0000 7f182ddbb700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18280b61d0 0x7f18280cf0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:41.800 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.790+0000 7f182ddbb700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1828142e90 0x7f18280cf630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:41.800 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.790+0000 7f182ddbb700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f182815de80 0x7f18281315d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:41.800 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.790+0000 7f182ddbb700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1828156b40 con 0x7f182815de80 2023-10-06T01:41:41.800 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.790+0000 7f182ddbb700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182815bb10 con 0x7f18280b61d0 2023-10-06T01:41:41.800 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.790+0000 7f182ddbb700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f182815b550 con 0x7f1828142e90 2023-10-06T01:41:41.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.790+0000 7f182d5ba700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f182815de80 0x7f18281315d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:41.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.791+0000 7f182d5ba700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f182815de80 0x7f18281315d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34318/0 (socket says 172.21.15.27:34318) 2023-10-06T01:41:41.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.791+0000 7f182d5ba700 1 -- 172.21.15.27:0/905321056 learned_addr learned my addr 172.21.15.27:0/905321056 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:41.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.791+0000 7f182cdb9700 1 --2- 172.21.15.27:0/905321056 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18280b61d0 0x7f18280cf0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:41.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.791+0000 7f1827fff700 1 --2- 172.21.15.27:0/905321056 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1828142e90 0x7f18280cf630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:41.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.791+0000 7f182d5ba700 1 -- 172.21.15.27:0/905321056 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1828142e90 msgr2=0x7f18280cf630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:41.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.791+0000 7f182d5ba700 1 --2- 172.21.15.27:0/905321056 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1828142e90 0x7f18280cf630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:41.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.791+0000 7f182d5ba700 1 -- 172.21.15.27:0/905321056 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18280b61d0 msgr2=0x7f18280cf0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:41.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.791+0000 7f182d5ba700 1 --2- 172.21.15.27:0/905321056 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18280b61d0 0x7f18280cf0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:41.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.791+0000 7f182d5ba700 1 -- 172.21.15.27:0/905321056 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18140094f0 con 0x7f182815de80 2023-10-06T01:41:41.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.791+0000 7f182d5ba700 1 --2- 172.21.15.27:0/905321056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f182815de80 0x7f18281315d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f18280b30b0 tx=0x7f1820019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:41.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.791+0000 7f1824ff9700 1 -- 172.21.15.27:0/905321056 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1820012e80 con 0x7f182815de80 2023-10-06T01:41:41.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.791+0000 7f1824ff9700 1 -- 172.21.15.27:0/905321056 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f182001d630 con 0x7f182815de80 2023-10-06T01:41:41.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.791+0000 7f182ddbb700 1 -- 172.21.15.27:0/905321056 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1820009a20 con 0x7f182815de80 2023-10-06T01:41:41.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.791+0000 7f1824ff9700 1 -- 172.21.15.27:0/905321056 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182001d9f0 con 0x7f182815de80 2023-10-06T01:41:41.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.791+0000 7f182ddbb700 1 -- 172.21.15.27:0/905321056 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18281567d0 con 0x7f182815de80 2023-10-06T01:41:41.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.792+0000 7f182ddbb700 1 -- 172.21.15.27:0/905321056 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1828156630 con 0x7f182815de80 2023-10-06T01:41:41.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.793+0000 7f1824ff9700 1 -- 172.21.15.27:0/905321056 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1820024070 con 0x7f182815de80 2023-10-06T01:41:41.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.793+0000 7f1824ff9700 1 --2- 172.21.15.27:0/905321056 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f180c066f30 0x7f180c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:41.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.793+0000 7f1824ff9700 1 -- 172.21.15.27:0/905321056 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f1820086ee0 con 0x7f182815de80 2023-10-06T01:41:41.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.793+0000 7f182cdb9700 1 --2- 172.21.15.27:0/905321056 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f180c066f30 0x7f180c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:41.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.794+0000 7f182cdb9700 1 --2- 172.21.15.27:0/905321056 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f180c066f30 0x7f180c0693e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f181c007fd0 tx=0x7f181c007d70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:41.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.798+0000 7f1824ff9700 1 -- 172.21.15.27:0/905321056 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1828156630 con 0x7f182815de80 2023-10-06T01:41:41.991 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.990+0000 7f182ddbb700 1 -- 172.21.15.27:0/905321056 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1828156630 con 0x7f182815de80 2023-10-06T01:41:41.991 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.990+0000 7f1824ff9700 1 -- 172.21.15.27:0/905321056 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1828156630 con 0x7f182815de80 2023-10-06T01:41:41.992 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2023-10-06T01:41:41.993 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.992+0000 7f182ddbb700 1 -- 172.21.15.27:0/905321056 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f180c066f30 msgr2=0x7f180c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:41.994 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.992+0000 7f182ddbb700 1 --2- 172.21.15.27:0/905321056 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f180c066f30 0x7f180c0693e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f181c007fd0 tx=0x7f181c007d70 comp rx=0 tx=0).stop 2023-10-06T01:41:41.994 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.992+0000 7f182ddbb700 1 -- 172.21.15.27:0/905321056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f182815de80 msgr2=0x7f18281315d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:41.994 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.992+0000 7f182ddbb700 1 --2- 172.21.15.27:0/905321056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f182815de80 0x7f18281315d0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f18280b30b0 tx=0x7f1820019000 comp rx=0 tx=0).stop 2023-10-06T01:41:41.994 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.992+0000 7f182ddbb700 1 -- 172.21.15.27:0/905321056 shutdown_connections 2023-10-06T01:41:41.994 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.992+0000 7f182ddbb700 1 --2- 172.21.15.27:0/905321056 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f180c066f30 0x7f180c0693e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:41.994 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.992+0000 7f182ddbb700 1 --2- 172.21.15.27:0/905321056 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f182815de80 0x7f18281315d0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:41.994 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.992+0000 7f182ddbb700 1 --2- 172.21.15.27:0/905321056 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f18280b61d0 0x7f18280cf0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:41.995 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.992+0000 7f182ddbb700 1 --2- 172.21.15.27:0/905321056 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1828142e90 0x7f18280cf630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:41.995 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.992+0000 7f182ddbb700 1 -- 172.21.15.27:0/905321056 >> 172.21.15.27:0/905321056 conn(0x7f18280ccfe0 msgr2=0x7f182815d220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:41.995 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.992+0000 7f182ddbb700 1 -- 172.21.15.27:0/905321056 shutdown_connections 2023-10-06T01:41:41.995 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:41.992+0000 7f182ddbb700 1 -- 172.21.15.27:0/905321056 wait complete. 2023-10-06T01:41:42.005 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-06T01:41:42.127 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.126+0000 7fbc62b91700 1 Processor -- start 2023-10-06T01:41:42.127 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.126+0000 7fbc62b91700 1 -- start start 2023-10-06T01:41:42.130 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.127+0000 7fbc62b91700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbc5c142e90 0x7fbc5c069810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:42.130 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.127+0000 7fbc62b91700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbc5c15d9a0 0x7fbc5c0b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:42.130 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.127+0000 7fbc62b91700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc5c069380 con 0x7fbc5c069d50 2023-10-06T01:41:42.130 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.127+0000 7fbc62b91700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc5c068770 con 0x7fbc5c15d9a0 2023-10-06T01:41:42.130 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.127+0000 7fbc62b91700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c068a50 con 0x7fbc5c142e90 2023-10-06T01:41:42.130 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.127+0000 7fbc62390700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbc5c15d9a0 0x7fbc5c0b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:42.130 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.127+0000 7fbc62390700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbc5c15d9a0 0x7fbc5c0b5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34328/0 (socket says 172.21.15.27:34328) 2023-10-06T01:41:42.130 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.127+0000 7fbc62390700 1 -- 172.21.15.27:0/2007867835 learned_addr learned my addr 172.21.15.27:0/2007867835 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:42.131 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.127+0000 7fbc61b8f700 1 --2- 172.21.15.27:0/2007867835 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbc5c142e90 0x7fbc5c069810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:42.131 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.127+0000 7fbc62390700 1 -- 172.21.15.27:0/2007867835 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbc5c142e90 msgr2=0x7fbc5c069810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:42.131 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.127+0000 7fbc62390700 1 --2- 172.21.15.27:0/2007867835 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbc5c142e90 0x7fbc5c069810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:42.131 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.127+0000 7fbc62390700 1 -- 172.21.15.27:0/2007867835 >> v1:172.21.15.124:6789/0 conn(0x7fbc5c069d50 legacy=0x7fbc5c15d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:42.131 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.127+0000 7fbc62390700 1 -- 172.21.15.27:0/2007867835 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc5c068d30 con 0x7fbc5c15d9a0 2023-10-06T01:41:42.131 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.127+0000 7fbc61b8f700 1 --2- 172.21.15.27:0/2007867835 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbc5c142e90 0x7fbc5c069810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:42.131 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.127+0000 7fbc62390700 1 --2- 172.21.15.27:0/2007867835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc5c15d9a0 0x7fbc5c0b5c90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c009f20 tx=0x7fbc4c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a14e3f4cab052443 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:42.131 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.128+0000 7fbc60b8d700 1 -- 172.21.15.27:0/2007867835 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc4c00e070 con 0x7fbc5c15d9a0 2023-10-06T01:41:42.131 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.128+0000 7fbc60b8d700 1 -- 172.21.15.27:0/2007867835 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc4c00ba80 con 0x7fbc5c15d9a0 2023-10-06T01:41:42.131 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.128+0000 7fbc60b8d700 1 -- 172.21.15.27:0/2007867835 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc4c0092e0 con 0x7fbc5c15d9a0 2023-10-06T01:41:42.132 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.128+0000 7fbc62b91700 1 -- 172.21.15.27:0/2007867835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc5c15d9a0 msgr2=0x7fbc5c0b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:42.132 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.128+0000 7fbc62b91700 1 --2- 172.21.15.27:0/2007867835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc5c15d9a0 0x7fbc5c0b5c90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c009f20 tx=0x7fbc4c009840 comp rx=0 tx=0).stop 2023-10-06T01:41:42.132 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.128+0000 7fbc62b91700 1 -- 172.21.15.27:0/2007867835 shutdown_connections 2023-10-06T01:41:42.132 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.128+0000 7fbc62b91700 1 --2- 172.21.15.27:0/2007867835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc5c15d9a0 0x7fbc5c0b5c90 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:42.132 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.128+0000 7fbc62b91700 1 --2- 172.21.15.27:0/2007867835 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbc5c142e90 0x7fbc5c069810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:42.132 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.128+0000 7fbc62b91700 1 -- 172.21.15.27:0/2007867835 >> 172.21.15.27:0/2007867835 conn(0x7fbc5c0ccfe0 msgr2=0x7fbc5c0cd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:42.132 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.128+0000 7fbc62b91700 1 -- 172.21.15.27:0/2007867835 shutdown_connections 2023-10-06T01:41:42.132 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.128+0000 7fbc62b91700 1 -- 172.21.15.27:0/2007867835 wait complete. 2023-10-06T01:41:42.135 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.129+0000 7fbc62b91700 1 Processor -- start 2023-10-06T01:41:42.135 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.129+0000 7fbc62b91700 1 -- start start 2023-10-06T01:41:42.135 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.130+0000 7fbc62b91700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc5c069d50 0x7fbc5c0c5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:42.135 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.130+0000 7fbc62b91700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbc5c142e90 0x7fbc5c0c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:42.135 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.130+0000 7fbc62b91700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbc5c15d9a0 0x7fbc5c13fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:42.135 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.130+0000 7fbc62b91700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c0cac70 con 0x7fbc5c069d50 2023-10-06T01:41:42.136 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.130+0000 7fbc62b91700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c0ca060 con 0x7fbc5c142e90 2023-10-06T01:41:42.136 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.130+0000 7fbc62b91700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c0ca340 con 0x7fbc5c15d9a0 2023-10-06T01:41:42.136 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.130+0000 7fbc61b8f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc5c069d50 0x7fbc5c0c5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:42.136 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.130+0000 7fbc61b8f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc5c069d50 0x7fbc5c0c5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34332/0 (socket says 172.21.15.27:34332) 2023-10-06T01:41:42.136 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.130+0000 7fbc61b8f700 1 -- 172.21.15.27:0/2364339906 learned_addr learned my addr 172.21.15.27:0/2364339906 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:42.136 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.130+0000 7fbc6138e700 1 --2- 172.21.15.27:0/2364339906 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbc5c142e90 0x7fbc5c0c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:42.136 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.130+0000 7fbc62390700 1 --2- 172.21.15.27:0/2364339906 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbc5c15d9a0 0x7fbc5c13fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:42.136 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.130+0000 7fbc61b8f700 1 -- 172.21.15.27:0/2364339906 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbc5c15d9a0 msgr2=0x7fbc5c13fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:42.136 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.130+0000 7fbc61b8f700 1 --2- 172.21.15.27:0/2364339906 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbc5c15d9a0 0x7fbc5c13fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:42.136 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.131+0000 7fbc61b8f700 1 -- 172.21.15.27:0/2364339906 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbc5c142e90 msgr2=0x7fbc5c0c97f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:42.136 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.131+0000 7fbc61b8f700 1 --2- 172.21.15.27:0/2364339906 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbc5c142e90 0x7fbc5c0c97f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:42.137 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.131+0000 7fbc61b8f700 1 -- 172.21.15.27:0/2364339906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc4c009ae0 con 0x7fbc5c069d50 2023-10-06T01:41:42.137 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.131+0000 7fbc62390700 1 --2- 172.21.15.27:0/2364339906 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbc5c15d9a0 0x7fbc5c13fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:42.137 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.131+0000 7fbc61b8f700 1 --2- 172.21.15.27:0/2364339906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc5c069d50 0x7fbc5c0c5310 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fbc5400e8d0 tx=0x7fbc54008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:42.137 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.131+0000 7fbc59ffb700 1 -- 172.21.15.27:0/2364339906 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc54013410 con 0x7fbc5c069d50 2023-10-06T01:41:42.137 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.131+0000 7fbc59ffb700 1 -- 172.21.15.27:0/2364339906 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc54013570 con 0x7fbc5c069d50 2023-10-06T01:41:42.137 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.131+0000 7fbc62b91700 1 -- 172.21.15.27:0/2364339906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc5c0ca620 con 0x7fbc5c069d50 2023-10-06T01:41:42.137 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.131+0000 7fbc59ffb700 1 -- 172.21.15.27:0/2364339906 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc54013910 con 0x7fbc5c069d50 2023-10-06T01:41:42.137 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.131+0000 7fbc62b91700 1 -- 172.21.15.27:0/2364339906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc5c0c9d30 con 0x7fbc5c069d50 2023-10-06T01:41:42.137 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.132+0000 7fbc62b91700 1 -- 172.21.15.27:0/2364339906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc5c0caf50 con 0x7fbc5c069d50 2023-10-06T01:41:42.137 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.132+0000 7fbc59ffb700 1 -- 172.21.15.27:0/2364339906 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fbc54013d00 con 0x7fbc5c069d50 2023-10-06T01:41:42.137 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.133+0000 7fbc59ffb700 1 --2- 172.21.15.27:0/2364339906 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fbc48066f30 0x7fbc480693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:42.137 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.133+0000 7fbc59ffb700 1 -- 172.21.15.27:0/2364339906 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fbc54084510 con 0x7fbc5c069d50 2023-10-06T01:41:42.138 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.133+0000 7fbc6138e700 1 --2- 172.21.15.27:0/2364339906 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fbc48066f30 0x7fbc480693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:42.138 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.134+0000 7fbc6138e700 1 --2- 172.21.15.27:0/2364339906 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fbc48066f30 0x7fbc480693e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fbc50003a80 tx=0x7fbc50007ec0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:42.138 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.137+0000 7fbc59ffb700 1 -- 172.21.15.27:0/2364339906 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbc54051580 con 0x7fbc5c069d50 2023-10-06T01:41:42.331 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.330+0000 7fbc62b91700 1 -- 172.21.15.27:0/2364339906 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbc5c04c3c0 con 0x7fbc5c069d50 2023-10-06T01:41:42.331 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.330+0000 7fbc59ffb700 1 -- 172.21.15.27:0/2364339906 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbc54054ba0 con 0x7fbc5c069d50 2023-10-06T01:41:42.332 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2023-10-06T01:41:42.333 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.332+0000 7fbc62b91700 1 -- 172.21.15.27:0/2364339906 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fbc48066f30 msgr2=0x7fbc480693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:42.333 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.332+0000 7fbc62b91700 1 --2- 172.21.15.27:0/2364339906 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fbc48066f30 0x7fbc480693e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fbc50003a80 tx=0x7fbc50007ec0 comp rx=0 tx=0).stop 2023-10-06T01:41:42.334 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.332+0000 7fbc62b91700 1 -- 172.21.15.27:0/2364339906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc5c069d50 msgr2=0x7fbc5c0c5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:42.334 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.332+0000 7fbc62b91700 1 --2- 172.21.15.27:0/2364339906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc5c069d50 0x7fbc5c0c5310 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fbc5400e8d0 tx=0x7fbc54008040 comp rx=0 tx=0).stop 2023-10-06T01:41:42.334 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.332+0000 7fbc62b91700 1 -- 172.21.15.27:0/2364339906 shutdown_connections 2023-10-06T01:41:42.334 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.332+0000 7fbc62b91700 1 --2- 172.21.15.27:0/2364339906 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fbc48066f30 0x7fbc480693e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:42.334 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.332+0000 7fbc62b91700 1 --2- 172.21.15.27:0/2364339906 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbc5c15d9a0 0x7fbc5c13fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:42.334 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.332+0000 7fbc62b91700 1 --2- 172.21.15.27:0/2364339906 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbc5c069d50 0x7fbc5c0c5310 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:42.334 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.332+0000 7fbc62b91700 1 --2- 172.21.15.27:0/2364339906 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbc5c142e90 0x7fbc5c0c97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:42.334 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.332+0000 7fbc62b91700 1 -- 172.21.15.27:0/2364339906 >> 172.21.15.27:0/2364339906 conn(0x7fbc5c0ccfe0 msgr2=0x7fbc5c141340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:42.334 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.332+0000 7fbc62b91700 1 -- 172.21.15.27:0/2364339906 shutdown_connections 2023-10-06T01:41:42.334 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.332+0000 7fbc62b91700 1 -- 172.21.15.27:0/2364339906 wait complete. 2023-10-06T01:41:42.345 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-06T01:41:42.467 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.465+0000 7f45eb3ab700 1 Processor -- start 2023-10-06T01:41:42.467 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.465+0000 7f45eb3ab700 1 -- start start 2023-10-06T01:41:42.470 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.466+0000 7f45eb3ab700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f45e4142e90 0x7f45e4069810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:42.470 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.466+0000 7f45eb3ab700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f45e4069d50 0x7f45e415d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:42.470 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.466+0000 7f45eb3ab700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f45e415de80 0x7f45e415d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:42.470 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.466+0000 7f45eb3ab700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45e4069380 con 0x7f45e415de80 2023-10-06T01:41:42.470 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.466+0000 7f45eb3ab700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45e4068770 con 0x7f45e4142e90 2023-10-06T01:41:42.471 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.466+0000 7f45eb3ab700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45e4068a50 con 0x7f45e4069d50 2023-10-06T01:41:42.471 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.466+0000 7f45eabaa700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f45e415de80 0x7f45e415d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:42.471 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.466+0000 7f45eabaa700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f45e415de80 0x7f45e415d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34348/0 (socket says 172.21.15.27:34348) 2023-10-06T01:41:42.471 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.466+0000 7f45eabaa700 1 -- 172.21.15.27:0/93279387 learned_addr learned my addr 172.21.15.27:0/93279387 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:42.471 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.466+0000 7f45ea3a9700 1 --2- 172.21.15.27:0/93279387 >> v2:172.21.15.124:3300/0 conn(0x7f45e4142e90 0x7f45e4069810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:42.471 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.466+0000 7f45e9ba8700 1 --2- 172.21.15.27:0/93279387 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f45e4069d50 0x7f45e415d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:42.471 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.466+0000 7f45eabaa700 1 -- 172.21.15.27:0/93279387 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f45e4069d50 msgr2=0x7f45e415d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:42.471 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.467+0000 7f45eabaa700 1 --2- 172.21.15.27:0/93279387 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f45e4069d50 0x7f45e415d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:42.472 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.467+0000 7f45eabaa700 1 -- 172.21.15.27:0/93279387 >> v2:172.21.15.124:3300/0 conn(0x7f45e4142e90 msgr2=0x7f45e4069810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:42.472 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.467+0000 7f45eabaa700 1 --2- 172.21.15.27:0/93279387 >> v2:172.21.15.124:3300/0 conn(0x7f45e4142e90 0x7f45e4069810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:42.472 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.467+0000 7f45eabaa700 1 -- 172.21.15.27:0/93279387 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e4068d30 con 0x7f45e415de80 2023-10-06T01:41:42.472 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.467+0000 7f45ea3a9700 1 --2- 172.21.15.27:0/93279387 >> v2:172.21.15.124:3300/0 conn(0x7f45e4142e90 0x7f45e4069810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:42.472 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.467+0000 7f45e9ba8700 1 --2- 172.21.15.27:0/93279387 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f45e4069d50 0x7f45e415d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:42.472 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.467+0000 7f45eabaa700 1 --2- 172.21.15.27:0/93279387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45e415de80 0x7f45e415d7d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f45e4089390 tx=0x7f45dc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2339390e76fa57d server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:42.472 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.467+0000 7f45e93a7700 1 -- 172.21.15.27:0/93279387 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45dc00e070 con 0x7f45e415de80 2023-10-06T01:41:42.472 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.467+0000 7f45e93a7700 1 -- 172.21.15.27:0/93279387 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45dc00ba50 con 0x7f45e415de80 2023-10-06T01:41:42.472 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.467+0000 7f45e93a7700 1 -- 172.21.15.27:0/93279387 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45dc0092e0 con 0x7f45e415de80 2023-10-06T01:41:42.472 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.467+0000 7f45eb3ab700 1 -- 172.21.15.27:0/93279387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45e415de80 msgr2=0x7f45e415d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:42.473 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.467+0000 7f45eb3ab700 1 --2- 172.21.15.27:0/93279387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45e415de80 0x7f45e415d7d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f45e4089390 tx=0x7f45dc009840 comp rx=0 tx=0).stop 2023-10-06T01:41:42.473 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.468+0000 7f45eb3ab700 1 -- 172.21.15.27:0/93279387 shutdown_connections 2023-10-06T01:41:42.473 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.468+0000 7f45eb3ab700 1 --2- 172.21.15.27:0/93279387 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45e415de80 0x7f45e415d7d0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:42.473 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.468+0000 7f45eb3ab700 1 --2- 172.21.15.27:0/93279387 >> v2:172.21.15.124:3300/0 conn(0x7f45e4142e90 0x7f45e4069810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:42.473 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.468+0000 7f45eb3ab700 1 --2- 172.21.15.27:0/93279387 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f45e4069d50 0x7f45e415d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:42.473 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.468+0000 7f45eb3ab700 1 -- 172.21.15.27:0/93279387 >> 172.21.15.27:0/93279387 conn(0x7f45e40ccfe0 msgr2=0x7f45e40cd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:42.473 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.468+0000 7f45eb3ab700 1 -- 172.21.15.27:0/93279387 shutdown_connections 2023-10-06T01:41:42.473 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.468+0000 7f45eb3ab700 1 -- 172.21.15.27:0/93279387 wait complete. 2023-10-06T01:41:42.474 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.469+0000 7f45eb3ab700 1 Processor -- start 2023-10-06T01:41:42.474 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.469+0000 7f45eb3ab700 1 -- start start 2023-10-06T01:41:42.476 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.469+0000 7f45eb3ab700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45e4069d50 0x7f45e4140000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:42.476 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.469+0000 7f45eb3ab700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f45e4142e90 0x7f45e40c6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:42.476 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.470+0000 7f45eb3ab700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f45e415de80 0x7f45e40cf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:42.476 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.470+0000 7f45eb3ab700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e4140540 con 0x7f45e4069d50 2023-10-06T01:41:42.477 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.470+0000 7f45eb3ab700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45e40c7d40 con 0x7f45e415de80 2023-10-06T01:41:42.477 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.470+0000 7f45eb3ab700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45e40c7130 con 0x7f45e4142e90 2023-10-06T01:41:42.477 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.470+0000 7f45ea3a9700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45e4069d50 0x7f45e4140000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:42.477 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.470+0000 7f45ea3a9700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45e4069d50 0x7f45e4140000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34364/0 (socket says 172.21.15.27:34364) 2023-10-06T01:41:42.477 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.470+0000 7f45ea3a9700 1 -- 172.21.15.27:0/3248466040 learned_addr learned my addr 172.21.15.27:0/3248466040 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:42.477 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.470+0000 7f45e9ba8700 1 --2- 172.21.15.27:0/3248466040 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f45e4142e90 0x7f45e40c6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:42.477 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.470+0000 7f45eabaa700 1 --2- 172.21.15.27:0/3248466040 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f45e415de80 0x7f45e40cf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:42.477 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.470+0000 7f45ea3a9700 1 -- 172.21.15.27:0/3248466040 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f45e4142e90 msgr2=0x7f45e40c6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:42.477 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.470+0000 7f45ea3a9700 1 --2- 172.21.15.27:0/3248466040 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f45e4142e90 0x7f45e40c6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:42.477 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.470+0000 7f45ea3a9700 1 -- 172.21.15.27:0/3248466040 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f45e415de80 msgr2=0x7f45e40cf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:42.477 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.470+0000 7f45ea3a9700 1 --2- 172.21.15.27:0/3248466040 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f45e415de80 0x7f45e40cf2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:42.477 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.470+0000 7f45ea3a9700 1 -- 172.21.15.27:0/3248466040 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45dc009ae0 con 0x7f45e4069d50 2023-10-06T01:41:42.478 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.470+0000 7f45eabaa700 1 --2- 172.21.15.27:0/3248466040 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f45e415de80 0x7f45e40cf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:42.478 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.470+0000 7f45ea3a9700 1 --2- 172.21.15.27:0/3248466040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45e4069d50 0x7f45e4140000 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f45e000e850 tx=0x7f45e0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:42.478 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.470+0000 7f45da7fc700 1 -- 172.21.15.27:0/3248466040 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e0013410 con 0x7f45e4069d50 2023-10-06T01:41:42.478 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.470+0000 7f45da7fc700 1 -- 172.21.15.27:0/3248466040 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45e0013570 con 0x7f45e4069d50 2023-10-06T01:41:42.478 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.470+0000 7f45eb3ab700 1 -- 172.21.15.27:0/3248466040 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45e40c7410 con 0x7f45e4069d50 2023-10-06T01:41:42.478 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.470+0000 7f45da7fc700 1 -- 172.21.15.27:0/3248466040 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e0013910 con 0x7f45e4069d50 2023-10-06T01:41:42.478 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.470+0000 7f45eb3ab700 1 -- 172.21.15.27:0/3248466040 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45e40c79d0 con 0x7f45e4069d50 2023-10-06T01:41:42.478 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.471+0000 7f45eb3ab700 1 -- 172.21.15.27:0/3248466040 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45e404c3c0 con 0x7f45e4069d50 2023-10-06T01:41:42.478 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.472+0000 7f45da7fc700 1 -- 172.21.15.27:0/3248466040 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f45e0013cc0 con 0x7f45e4069d50 2023-10-06T01:41:42.479 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.472+0000 7f45da7fc700 1 --2- 172.21.15.27:0/3248466040 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f45c8066f30 0x7f45c80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:42.479 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.472+0000 7f45da7fc700 1 -- 172.21.15.27:0/3248466040 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f45e0084320 con 0x7f45e4069d50 2023-10-06T01:41:42.479 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.472+0000 7f45e9ba8700 1 --2- 172.21.15.27:0/3248466040 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f45c8066f30 0x7f45c80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:42.479 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.473+0000 7f45e9ba8700 1 --2- 172.21.15.27:0/3248466040 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f45c8066f30 0x7f45c80693e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f45e4141ca0 tx=0x7f45d4008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:42.479 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.477+0000 7f45da7fc700 1 -- 172.21.15.27:0/3248466040 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f45e0051430 con 0x7f45e4069d50 2023-10-06T01:41:42.673 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.672+0000 7f45eb3ab700 1 -- 172.21.15.27:0/3248466040 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f45e40c8020 con 0x7f45e4069d50 2023-10-06T01:41:42.674 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.673+0000 7f45da7fc700 1 -- 172.21.15.27:0/3248466040 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f45e0054a50 con 0x7f45e4069d50 2023-10-06T01:41:42.674 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2023-10-06T01:41:42.676 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.675+0000 7f45eb3ab700 1 -- 172.21.15.27:0/3248466040 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f45c8066f30 msgr2=0x7f45c80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:42.676 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.675+0000 7f45eb3ab700 1 --2- 172.21.15.27:0/3248466040 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f45c8066f30 0x7f45c80693e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f45e4141ca0 tx=0x7f45d4008040 comp rx=0 tx=0).stop 2023-10-06T01:41:42.676 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.675+0000 7f45eb3ab700 1 -- 172.21.15.27:0/3248466040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45e4069d50 msgr2=0x7f45e4140000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.675+0000 7f45eb3ab700 1 --2- 172.21.15.27:0/3248466040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45e4069d50 0x7f45e4140000 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f45e000e850 tx=0x7f45e0008040 comp rx=0 tx=0).stop 2023-10-06T01:41:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.675+0000 7f45eb3ab700 1 -- 172.21.15.27:0/3248466040 shutdown_connections 2023-10-06T01:41:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.675+0000 7f45eb3ab700 1 --2- 172.21.15.27:0/3248466040 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f45c8066f30 0x7f45c80693e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.675+0000 7f45eb3ab700 1 --2- 172.21.15.27:0/3248466040 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f45e415de80 0x7f45e40cf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.675+0000 7f45eb3ab700 1 --2- 172.21.15.27:0/3248466040 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f45e4069d50 0x7f45e4140000 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.675+0000 7f45eb3ab700 1 --2- 172.21.15.27:0/3248466040 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f45e4142e90 0x7f45e40c6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.675+0000 7f45eb3ab700 1 -- 172.21.15.27:0/3248466040 >> 172.21.15.27:0/3248466040 conn(0x7f45e40ccfe0 msgr2=0x7f45e40b5510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.675+0000 7f45eb3ab700 1 -- 172.21.15.27:0/3248466040 shutdown_connections 2023-10-06T01:41:42.677 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.675+0000 7f45eb3ab700 1 -- 172.21.15.27:0/3248466040 wait complete. 2023-10-06T01:41:42.688 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T01:41:42.807 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.805+0000 7f864aade700 1 Processor -- start 2023-10-06T01:41:42.807 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.805+0000 7f864aade700 1 -- start start 2023-10-06T01:41:42.809 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.806+0000 7f864aade700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f86440b5600 0x7f86440d0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:42.810 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.806+0000 7f864aade700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86440b2ad0 con 0x7f864415d560 2023-10-06T01:41:42.810 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.806+0000 7f864aade700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86440b4160 con 0x7f864415d930 2023-10-06T01:41:42.810 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.806+0000 7f864aade700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86440b2230 con 0x7f86440b5600 2023-10-06T01:41:42.810 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.806+0000 7f864907b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f86440b5600 0x7f86440d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:42.810 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.806+0000 7f864887a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f864415d560 0x7f86440b1a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41046/0 (socket says 172.21.15.27:41046) 2023-10-06T01:41:42.810 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.806+0000 7f864887a700 1 -- 172.21.15.27:0/331854769 learned_addr learned my addr 172.21.15.27:0/331854769 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:42.810 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.807+0000 7f864907b700 1 -- 172.21.15.27:0/331854769 >> v1:172.21.15.124:6789/0 conn(0x7f864415d930 legacy=0x7f86441591b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:42.810 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.807+0000 7f864907b700 1 -- 172.21.15.27:0/331854769 >> v1:172.21.15.27:6789/0 conn(0x7f864415d560 legacy=0x7f86440b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:42.811 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.807+0000 7f864907b700 1 -- 172.21.15.27:0/331854769 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86440b2510 con 0x7f86440b5600 2023-10-06T01:41:42.811 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.807+0000 7f86437fe700 1 -- 172.21.15.27:0/331854769 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820782377 0 0) 0x7f86440b2ad0 con 0x7f864415d560 2023-10-06T01:41:42.811 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.807+0000 7f864907b700 1 --2- 172.21.15.27:0/331854769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86440b5600 0x7f86440d0030 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f862c0098e0 tx=0x7f862c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67fb17c48bc01f34 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:42.811 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.807+0000 7f86437fe700 1 -- 172.21.15.27:0/331854769 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f862c009e80 con 0x7f86440b5600 2023-10-06T01:41:42.811 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.807+0000 7f86437fe700 1 -- 172.21.15.27:0/331854769 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f862c00b860 con 0x7f86440b5600 2023-10-06T01:41:42.811 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.807+0000 7f86437fe700 1 -- 172.21.15.27:0/331854769 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f862c00bbe0 con 0x7f86440b5600 2023-10-06T01:41:42.811 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.808+0000 7f864aade700 1 -- 172.21.15.27:0/331854769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86440b5600 msgr2=0x7f86440d0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:42.811 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.808+0000 7f864aade700 1 --2- 172.21.15.27:0/331854769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86440b5600 0x7f86440d0030 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f862c0098e0 tx=0x7f862c0091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:42.811 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.808+0000 7f864aade700 1 -- 172.21.15.27:0/331854769 shutdown_connections 2023-10-06T01:41:42.811 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.808+0000 7f864aade700 1 --2- 172.21.15.27:0/331854769 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86440b5600 0x7f86440d0030 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:42.812 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.808+0000 7f864aade700 1 -- 172.21.15.27:0/331854769 >> 172.21.15.27:0/331854769 conn(0x7f86440d1180 msgr2=0x7f8644069640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:42.812 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.808+0000 7f864aade700 1 -- 172.21.15.27:0/331854769 shutdown_connections 2023-10-06T01:41:42.812 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.808+0000 7f864aade700 1 -- 172.21.15.27:0/331854769 wait complete. 2023-10-06T01:41:42.815 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.809+0000 7f864aade700 1 Processor -- start 2023-10-06T01:41:42.815 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.809+0000 7f864aade700 1 -- start start 2023-10-06T01:41:42.815 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.810+0000 7f864aade700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86440b5600 0x7f8644153400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:42.815 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.810+0000 7f864aade700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f864415d560 0x7f8644150500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:42.815 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.810+0000 7f864aade700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f864415d930 0x7f864415a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:42.815 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.810+0000 7f864aade700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8644152f70 con 0x7f86440b5600 2023-10-06T01:41:42.815 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.810+0000 7f864aade700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8644153940 con 0x7f864415d930 2023-10-06T01:41:42.815 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.810+0000 7f864aade700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8644152360 con 0x7f864415d560 2023-10-06T01:41:42.815 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.810+0000 7f864887a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86440b5600 0x7f8644153400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:42.815 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.810+0000 7f864887a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86440b5600 0x7f8644153400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34384/0 (socket says 172.21.15.27:34384) 2023-10-06T01:41:42.815 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.810+0000 7f864887a700 1 -- 172.21.15.27:0/2353964422 learned_addr learned my addr 172.21.15.27:0/2353964422 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:42.816 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.810+0000 7f864907b700 1 --2- 172.21.15.27:0/2353964422 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f864415d930 0x7f864415a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:42.816 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.810+0000 7f8643fff700 1 --2- 172.21.15.27:0/2353964422 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f864415d560 0x7f8644150500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:42.816 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.810+0000 7f864887a700 1 -- 172.21.15.27:0/2353964422 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f864415d560 msgr2=0x7f8644150500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:42.816 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.810+0000 7f864887a700 1 --2- 172.21.15.27:0/2353964422 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f864415d560 0x7f8644150500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:42.816 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.810+0000 7f864887a700 1 -- 172.21.15.27:0/2353964422 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f864415d930 msgr2=0x7f864415a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:42.816 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.810+0000 7f864887a700 1 --2- 172.21.15.27:0/2353964422 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f864415d930 0x7f864415a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:42.816 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.810+0000 7f864887a700 1 -- 172.21.15.27:0/2353964422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f862c009420 con 0x7f86440b5600 2023-10-06T01:41:42.816 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.810+0000 7f864887a700 1 --2- 172.21.15.27:0/2353964422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86440b5600 0x7f8644153400 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f863c00ccc0 tx=0x7f863c001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:42.816 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.810+0000 7f8640ff9700 1 -- 172.21.15.27:0/2353964422 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863c00a8e0 con 0x7f86440b5600 2023-10-06T01:41:42.816 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.810+0000 7f8640ff9700 1 -- 172.21.15.27:0/2353964422 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f863c00aa40 con 0x7f86440b5600 2023-10-06T01:41:42.816 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.810+0000 7f864aade700 1 -- 172.21.15.27:0/2353964422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8644152640 con 0x7f86440b5600 2023-10-06T01:41:42.817 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.810+0000 7f8640ff9700 1 -- 172.21.15.27:0/2353964422 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863c00ae00 con 0x7f86440b5600 2023-10-06T01:41:42.817 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.810+0000 7f864aade700 1 -- 172.21.15.27:0/2353964422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8644152c00 con 0x7f86440b5600 2023-10-06T01:41:42.817 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.811+0000 7f864aade700 1 -- 172.21.15.27:0/2353964422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8644153250 con 0x7f86440b5600 2023-10-06T01:41:42.817 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.812+0000 7f8640ff9700 1 -- 172.21.15.27:0/2353964422 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f863c007300 con 0x7f86440b5600 2023-10-06T01:41:42.817 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.812+0000 7f8640ff9700 1 --2- 172.21.15.27:0/2353964422 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8624066f30 0x7f86240693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:42.817 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.812+0000 7f8640ff9700 1 -- 172.21.15.27:0/2353964422 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f863c084a10 con 0x7f86440b5600 2023-10-06T01:41:42.817 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.812+0000 7f8643fff700 1 --2- 172.21.15.27:0/2353964422 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8624066f30 0x7f86240693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:42.817 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.813+0000 7f8643fff700 1 --2- 172.21.15.27:0/2353964422 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8624066f30 0x7f86240693e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f864414d610 tx=0x7f8634007ec0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:42.818 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:42.816+0000 7f8640ff9700 1 -- 172.21.15.27:0/2353964422 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f863c051ba0 con 0x7f86440b5600 2023-10-06T01:41:43.010 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.009+0000 7f864aade700 1 -- 172.21.15.27:0/2353964422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8644064050 con 0x7f86440b5600 2023-10-06T01:41:43.011 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.010+0000 7f8640ff9700 1 -- 172.21.15.27:0/2353964422 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16866 (secure 0 0 0) 0x7f863c0551c0 con 0x7f86440b5600 2023-10-06T01:41:43.012 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:43.012 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":20,"fsid":"20a7bff0-74f8-4f2f-b6c6-899e804bc965","created":"2023-10-06T01:40:54.474215+0000","modified":"2023-10-06T01:41:14.324546+0000","last_up_change":"2023-10-06T01:41:05.242847+0000","last_in_change":"2023-10-06T01:40:56.890960+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-10-06T01:41:05.685386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-10-06T01:41:06.092995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T01:41:10.434257+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T01:41:11.700262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b902c86d-800c-40c5-b152-b545e50d02c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":90644},{"type":"v1","addr":"172.21.15.27:6801","nonce":90644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":90644},{"type":"v1","addr":"172.21.15.27:6803","nonce":90644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":90644},{"type":"v1","addr":"172.21.15.27:6807","nonce":90644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":90644},{"type":"v1","addr":"172.21.15.27:6805","nonce":90644}]},"public_addr":"172.21.15.27:6801/90644","cluster_addr":"172.21.15.27:6803/90644","heartbeat_back_addr":"172.21.15.27:6807/90644","heartbeat_front_addr":"172.21.15.27:6805/90644","state":["exists","up"]},{"osd":1,"uuid":"5b4a70e5-b06e-48cd-bfc4-7962c5d05386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":90643},{"type":"v1","addr":"172.21.15.27:6809","nonce":90643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":90643},{"type":"v1","addr":"172.21.15.27:6811","nonce":90643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":90643},{"type":"v1","addr":"172.21.15.27:6815","nonce":90643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":90643},{"type":"v1","addr":"172.21.15.27:6813","nonce":90643}]},"public_addr":"172.21.15.27:6809/90643","cluster_addr":"172.21.15.27:6811/90643","heartbeat_back_addr":"172.21.15.27:6815/90643","heartbeat_front_addr":"172.21.15.27:6813/90643","state":["exists","up"]},{"osd":2,"uuid":"22d9de46-4073-4336-9122-5227e601eca0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":90645},{"type":"v1","addr":"172.21.15.27:6817","nonce":90645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":90645},{"type":"v1","addr":"172.21.15.27:6819","nonce":90645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":90645},{"type":"v1","addr":"172.21.15.27:6823","nonce":90645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":90645},{"type":"v1","addr":"172.21.15.27:6821","nonce":90645}]},"public_addr":"172.21.15.27:6817/90645","cluster_addr":"172.21.15.27:6819/90645","heartbeat_back_addr":"172.21.15.27:6823/90645","heartbeat_front_addr":"172.21.15.27:6821/90645","state":["exists","up"]},{"osd":3,"uuid":"438bb2d3-34e2-4b7c-9734-8360c1b67650","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":90668},{"type":"v1","addr":"172.21.15.27:6825","nonce":90668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":90668},{"type":"v1","addr":"172.21.15.27:6827","nonce":90668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":90668},{"type":"v1","addr":"172.21.15.27:6831","nonce":90668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":90668},{"type":"v1","addr":"172.21.15.27:6829","nonce":90668}]},"public_addr":"172.21.15.27:6825/90668","cluster_addr":"172.21.15.27:6827/90668","heartbeat_back_addr":"172.21.15.27:6831/90668","heartbeat_front_addr":"172.21.15.27:6829/90668","state":["exists","up"]},{"osd":4,"uuid":"e1956a8e-e0f8-4e7c-acbb-40037415a354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":90863},{"type":"v1","addr":"172.21.15.124:6801","nonce":90863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":90863},{"type":"v1","addr":"172.21.15.124:6803","nonce":90863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":90863},{"type":"v1","addr":"172.21.15.124:6807","nonce":90863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":90863},{"type":"v1","addr":"172.21.15.124:6805","nonce":90863}]},"public_addr":"172.21.15.124:6801/90863","cluster_addr":"172.21.15.124:6803/90863","heartbeat_back_addr":"172.21.15.124:6807/90863","heartbeat_front_addr":"172.21.15.124:6805/90863","state":["exists","up"]},{"osd":5,"uuid":"306c59d4-846d-4b73-b1c9-051bec9e9368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":90885},{"type":"v1","addr":"172.21.15.124:6817","nonce":90885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":90885},{"type":"v1","addr":"172.21.15.124:6819","nonce":90885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":90885},{"type":"v1","addr":"172.21.15.124:6823","nonce":90885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":90885},{"type":"v1","addr":"172.21.15.124:6821","nonce":90885}]},"public_addr":"172.21.15.124:6817/90885","cluster_addr":"172.21.15.124:6819/90885","heartbeat_back_addr":"172.21.15.124:6823/90885","heartbeat_front_addr":"172.21.15.124:6821/90885","state":["exists","up"]},{"osd":6,"uuid":"15463e53-ac8d-47d1-b4f6-c0d31f29e4a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":90864},{"type":"v1","addr":"172.21.15.124:6809","nonce":90864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":90864},{"type":"v1","addr":"172.21.15.124:6811","nonce":90864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":90864},{"type":"v1","addr":"172.21.15.124:6815","nonce":90864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":90864},{"type":"v1","addr":"172.21.15.124:6813","nonce":90864}]},"public_addr":"172.21.15.124:6809/90864","cluster_addr":"172.21.15.124:6811/90864","heartbeat_back_addr":"172.21.15.124:6815/90864","heartbeat_front_addr":"172.21.15.124:6813/90864","state":["exists","up"]},{"osd":7,"uuid":"ae463a90-a9d3-498f-86fa-47a35b3fe889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":90894},{"type":"v1","addr":"172.21.15.124:6825","nonce":90894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":90894},{"type":"v1","addr":"172.21.15.124:6827","nonce":90894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":90894},{"type":"v1","addr":"172.21.15.124:6831","nonce":90894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":90894},{"type":"v1","addr":"172.21.15.124:6829","nonce":90894}]},"public_addr":"172.21.15.124:6825/90894","cluster_addr":"172.21.15.124:6827/90894","heartbeat_back_addr":"172.21.15.124:6831/90894","heartbeat_front_addr":"172.21.15.124:6829/90894","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.827704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.839951+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.032230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.050605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.987908+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.031290+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.080707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.053692+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T01:41:43.014 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.012+0000 7f864aade700 1 -- 172.21.15.27:0/2353964422 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8624066f30 msgr2=0x7f86240693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:43.014 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.012+0000 7f864aade700 1 --2- 172.21.15.27:0/2353964422 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8624066f30 0x7f86240693e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f864414d610 tx=0x7f8634007ec0 comp rx=0 tx=0).stop 2023-10-06T01:41:43.014 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.012+0000 7f864aade700 1 -- 172.21.15.27:0/2353964422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86440b5600 msgr2=0x7f8644153400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:43.014 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.012+0000 7f864aade700 1 --2- 172.21.15.27:0/2353964422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86440b5600 0x7f8644153400 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f863c00ccc0 tx=0x7f863c001a80 comp rx=0 tx=0).stop 2023-10-06T01:41:43.014 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.012+0000 7f864aade700 1 -- 172.21.15.27:0/2353964422 shutdown_connections 2023-10-06T01:41:43.015 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.012+0000 7f864aade700 1 --2- 172.21.15.27:0/2353964422 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8624066f30 0x7f86240693e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:43.015 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.012+0000 7f864aade700 1 --2- 172.21.15.27:0/2353964422 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f864415d930 0x7f864415a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:43.015 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.012+0000 7f864aade700 1 --2- 172.21.15.27:0/2353964422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f86440b5600 0x7f8644153400 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:43.015 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.012+0000 7f864aade700 1 --2- 172.21.15.27:0/2353964422 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f864415d560 0x7f8644150500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:43.015 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.012+0000 7f864aade700 1 -- 172.21.15.27:0/2353964422 >> 172.21.15.27:0/2353964422 conn(0x7f86440d1180 msgr2=0x7f86440cec40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:43.015 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.012+0000 7f864aade700 1 -- 172.21.15.27:0/2353964422 shutdown_connections 2023-10-06T01:41:43.015 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.012+0000 7f864aade700 1 -- 172.21.15.27:0/2353964422 wait complete. 2023-10-06T01:41:43.026 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-10-06T01:41:05.685386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-10-06T01:41:06.092995+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-06T01:41:10.434257+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-10-06T01:41:11.700262+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-06T01:41:43.026 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-06T01:41:43.147 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.146+0000 7fd329a7d700 1 Processor -- start 2023-10-06T01:41:43.148 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.146+0000 7fd329a7d700 1 -- start start 2023-10-06T01:41:43.150 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.147+0000 7fd329a7d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd324142e90 0x7fd324069810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:43.150 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.147+0000 7fd329a7d700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fd32415d9a0 0x7fd3240b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:43.150 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.147+0000 7fd329a7d700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd324069380 con 0x7fd324069d50 2023-10-06T01:41:43.150 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.147+0000 7fd329a7d700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd324068770 con 0x7fd32415d9a0 2023-10-06T01:41:43.150 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.147+0000 7fd329a7d700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd324068a50 con 0x7fd324142e90 2023-10-06T01:41:43.150 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.147+0000 7fd32927c700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fd32415d9a0 0x7fd3240b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:43.150 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.147+0000 7fd328a7b700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd324142e90 0x7fd324069810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.147+0000 7fd323fff700 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fd324069d50 0x7fd32415d290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.27:56000/0 (socket says 172.21.15.27:56000) 2023-10-06T01:41:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.147+0000 7fd323fff700 1 -- 172.21.15.27:0/2694887078 learned_addr learned my addr 172.21.15.27:0/2694887078 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.148+0000 7fd328a7b700 1 -- 172.21.15.27:0/2694887078 >> v2:172.21.15.124:3300/0 conn(0x7fd32415d9a0 msgr2=0x7fd3240b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.148+0000 7fd328a7b700 1 --2- 172.21.15.27:0/2694887078 >> v2:172.21.15.124:3300/0 conn(0x7fd32415d9a0 0x7fd3240b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.148+0000 7fd328a7b700 1 -- 172.21.15.27:0/2694887078 >> v1:172.21.15.124:6789/0 conn(0x7fd324069d50 legacy=0x7fd32415d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.148+0000 7fd328a7b700 1 -- 172.21.15.27:0/2694887078 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd324068d30 con 0x7fd324142e90 2023-10-06T01:41:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.148+0000 7fd32927c700 1 --2- 172.21.15.27:0/2694887078 >> v2:172.21.15.124:3300/0 conn(0x7fd32415d9a0 0x7fd3240b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:41:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.148+0000 7fd328a7b700 1 --2- 172.21.15.27:0/2694887078 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd324142e90 0x7fd324069810 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd31c0098e0 tx=0x7fd31c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57ace659d1dc048c server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.148+0000 7fd3237fe700 1 -- 172.21.15.27:0/2694887078 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd31c00b640 con 0x7fd324142e90 2023-10-06T01:41:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.148+0000 7fd3237fe700 1 -- 172.21.15.27:0/2694887078 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd31c00b7a0 con 0x7fd324142e90 2023-10-06T01:41:43.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.148+0000 7fd3237fe700 1 -- 172.21.15.27:0/2694887078 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd31c00bb40 con 0x7fd324142e90 2023-10-06T01:41:43.152 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.148+0000 7fd329a7d700 1 -- 172.21.15.27:0/2694887078 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd324142e90 msgr2=0x7fd324069810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:43.152 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.148+0000 7fd329a7d700 1 --2- 172.21.15.27:0/2694887078 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd324142e90 0x7fd324069810 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd31c0098e0 tx=0x7fd31c0091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:43.152 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.148+0000 7fd329a7d700 1 -- 172.21.15.27:0/2694887078 shutdown_connections 2023-10-06T01:41:43.152 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.148+0000 7fd329a7d700 1 --2- 172.21.15.27:0/2694887078 >> v2:172.21.15.124:3300/0 conn(0x7fd32415d9a0 0x7fd3240b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:43.152 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.148+0000 7fd329a7d700 1 --2- 172.21.15.27:0/2694887078 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd324142e90 0x7fd324069810 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:43.152 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.149+0000 7fd329a7d700 1 -- 172.21.15.27:0/2694887078 >> 172.21.15.27:0/2694887078 conn(0x7fd3240ccfe0 msgr2=0x7fd3240cd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:43.152 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.149+0000 7fd329a7d700 1 -- 172.21.15.27:0/2694887078 shutdown_connections 2023-10-06T01:41:43.152 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.149+0000 7fd329a7d700 1 -- 172.21.15.27:0/2694887078 wait complete. 2023-10-06T01:41:43.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.150+0000 7fd329a7d700 1 Processor -- start 2023-10-06T01:41:43.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.150+0000 7fd329a7d700 1 -- start start 2023-10-06T01:41:43.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.150+0000 7fd329a7d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd324069d50 0x7fd32415b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:43.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.150+0000 7fd329a7d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd324142e90 0x7fd32413cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:43.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.150+0000 7fd329a7d700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd32415d9a0 0x7fd32413d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:43.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.150+0000 7fd329a7d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd32415b400 con 0x7fd324069d50 2023-10-06T01:41:43.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.150+0000 7fd329a7d700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd32415a7f0 con 0x7fd32415d9a0 2023-10-06T01:41:43.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.150+0000 7fd329a7d700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd32415aad0 con 0x7fd324142e90 2023-10-06T01:41:43.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.151+0000 7fd328a7b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd324069d50 0x7fd32415b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:43.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.151+0000 7fd328a7b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd324069d50 0x7fd32415b890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34390/0 (socket says 172.21.15.27:34390) 2023-10-06T01:41:43.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.151+0000 7fd328a7b700 1 -- 172.21.15.27:0/3306616078 learned_addr learned my addr 172.21.15.27:0/3306616078 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:43.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.151+0000 7fd323fff700 1 --2- 172.21.15.27:0/3306616078 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd324142e90 0x7fd32413cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:43.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.151+0000 7fd32927c700 1 --2- 172.21.15.27:0/3306616078 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd32415d9a0 0x7fd32413d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:43.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.151+0000 7fd328a7b700 1 -- 172.21.15.27:0/3306616078 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd324142e90 msgr2=0x7fd32413cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:43.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.151+0000 7fd328a7b700 1 --2- 172.21.15.27:0/3306616078 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd324142e90 0x7fd32413cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:43.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.151+0000 7fd328a7b700 1 -- 172.21.15.27:0/3306616078 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd32415d9a0 msgr2=0x7fd32413d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:43.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.151+0000 7fd328a7b700 1 --2- 172.21.15.27:0/3306616078 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd32415d9a0 0x7fd32413d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:43.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.151+0000 7fd328a7b700 1 -- 172.21.15.27:0/3306616078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd30c009420 con 0x7fd324069d50 2023-10-06T01:41:43.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.151+0000 7fd328a7b700 1 --2- 172.21.15.27:0/3306616078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd324069d50 0x7fd32415b890 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd324068b90 tx=0x7fd31c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:43.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.151+0000 7fd320ff9700 1 -- 172.21.15.27:0/3306616078 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd31c01bc20 con 0x7fd324069d50 2023-10-06T01:41:43.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.151+0000 7fd320ff9700 1 -- 172.21.15.27:0/3306616078 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd31c01bd80 con 0x7fd324069d50 2023-10-06T01:41:43.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.151+0000 7fd329a7d700 1 -- 172.21.15.27:0/3306616078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd31c009420 con 0x7fd324069d50 2023-10-06T01:41:43.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.151+0000 7fd320ff9700 1 -- 172.21.15.27:0/3306616078 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd31c011670 con 0x7fd324069d50 2023-10-06T01:41:43.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.151+0000 7fd329a7d700 1 -- 172.21.15.27:0/3306616078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd32415b090 con 0x7fd324069d50 2023-10-06T01:41:43.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.152+0000 7fd329a7d700 1 -- 172.21.15.27:0/3306616078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd32404c3c0 con 0x7fd324069d50 2023-10-06T01:41:43.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.153+0000 7fd320ff9700 1 -- 172.21.15.27:0/3306616078 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd31c011a20 con 0x7fd324069d50 2023-10-06T01:41:43.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.153+0000 7fd320ff9700 1 --2- 172.21.15.27:0/3306616078 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd304066fb0 0x7fd304069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:43.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.153+0000 7fd320ff9700 1 -- 172.21.15.27:0/3306616078 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fd31c085b20 con 0x7fd324069d50 2023-10-06T01:41:43.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.153+0000 7fd323fff700 1 --2- 172.21.15.27:0/3306616078 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd304066fb0 0x7fd304069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:43.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.154+0000 7fd323fff700 1 --2- 172.21.15.27:0/3306616078 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd304066fb0 0x7fd304069460 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd32413f700 tx=0x7fd314001a10 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:43.158 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.157+0000 7fd320ff9700 1 -- 172.21.15.27:0/3306616078 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd31c087e60 con 0x7fd324069d50 2023-10-06T01:41:43.351 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.350+0000 7fd329a7d700 1 -- 172.21.15.27:0/3306616078 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd324069010 con 0x7fd324069d50 2023-10-06T01:41:43.351 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.350+0000 7fd320ff9700 1 -- 172.21.15.27:0/3306616078 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd31c052c50 con 0x7fd324069d50 2023-10-06T01:41:43.352 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 8 2023-10-06T01:41:43.354 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.352+0000 7fd329a7d700 1 -- 172.21.15.27:0/3306616078 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd304066fb0 msgr2=0x7fd304069460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:43.354 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.352+0000 7fd329a7d700 1 --2- 172.21.15.27:0/3306616078 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd304066fb0 0x7fd304069460 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd32413f700 tx=0x7fd314001a10 comp rx=0 tx=0).stop 2023-10-06T01:41:43.354 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.352+0000 7fd329a7d700 1 -- 172.21.15.27:0/3306616078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd324069d50 msgr2=0x7fd32415b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:43.354 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.352+0000 7fd329a7d700 1 --2- 172.21.15.27:0/3306616078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd324069d50 0x7fd32415b890 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd324068b90 tx=0x7fd31c017000 comp rx=0 tx=0).stop 2023-10-06T01:41:43.354 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.352+0000 7fd329a7d700 1 -- 172.21.15.27:0/3306616078 shutdown_connections 2023-10-06T01:41:43.354 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.352+0000 7fd329a7d700 1 --2- 172.21.15.27:0/3306616078 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd304066fb0 0x7fd304069460 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:43.354 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.352+0000 7fd329a7d700 1 --2- 172.21.15.27:0/3306616078 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd32415d9a0 0x7fd32413d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:43.354 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.352+0000 7fd329a7d700 1 --2- 172.21.15.27:0/3306616078 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd324069d50 0x7fd32415b890 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:43.355 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.352+0000 7fd329a7d700 1 --2- 172.21.15.27:0/3306616078 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd324142e90 0x7fd32413cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:43.355 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.352+0000 7fd329a7d700 1 -- 172.21.15.27:0/3306616078 >> 172.21.15.27:0/3306616078 conn(0x7fd3240ccfe0 msgr2=0x7fd3240b4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:43.355 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.352+0000 7fd329a7d700 1 -- 172.21.15.27:0/3306616078 shutdown_connections 2023-10-06T01:41:43.355 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.352+0000 7fd329a7d700 1 -- 172.21.15.27:0/3306616078 wait complete. 2023-10-06T01:41:43.365 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-06T01:41:43.488 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.487+0000 7f7738c4a700 1 Processor -- start 2023-10-06T01:41:43.488 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.487+0000 7f7738c4a700 1 -- start start 2023-10-06T01:41:43.491 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.488+0000 7f7738c4a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7734142e90 0x7f7734069810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:43.491 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.488+0000 7f7738c4a700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f7734069d50 0x7f773415d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:43.491 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.488+0000 7f7738c4a700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7734069380 con 0x7f773415de80 2023-10-06T01:41:43.491 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.488+0000 7f7738c4a700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7734068770 con 0x7f7734142e90 2023-10-06T01:41:43.491 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.488+0000 7f7738c4a700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7734068a50 con 0x7f7734069d50 2023-10-06T01:41:43.491 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.488+0000 7f77337fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7734142e90 0x7f7734069810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:43.491 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.488+0000 7f77337fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7734142e90 0x7f7734069810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34400/0 (socket says 172.21.15.27:34400) 2023-10-06T01:41:43.492 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.488+0000 7f77337fe700 1 -- 172.21.15.27:0/1178988490 learned_addr learned my addr 172.21.15.27:0/1178988490 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:43.492 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.488+0000 7f7732ffd700 1 --2- 172.21.15.27:0/1178988490 >> v2:172.21.15.124:3300/0 conn(0x7f7734069d50 0x7f773415d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:43.492 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.488+0000 7f77337fe700 1 -- 172.21.15.27:0/1178988490 >> v2:172.21.15.124:3300/0 conn(0x7f7734069d50 msgr2=0x7f773415d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:43.492 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.488+0000 7f77337fe700 1 --2- 172.21.15.27:0/1178988490 >> v2:172.21.15.124:3300/0 conn(0x7f7734069d50 0x7f773415d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:43.492 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.488+0000 7f77337fe700 1 -- 172.21.15.27:0/1178988490 >> v1:172.21.15.124:6789/0 conn(0x7f773415de80 legacy=0x7f77340b0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:43.492 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.488+0000 7f77337fe700 1 -- 172.21.15.27:0/1178988490 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7734068d30 con 0x7f7734142e90 2023-10-06T01:41:43.492 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.488+0000 7f77337fe700 1 --2- 172.21.15.27:0/1178988490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7734142e90 0x7f7734069810 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f773406fd60 tx=0x7f7724009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=212a1f21241b87db server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:43.492 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.488+0000 7f77327fc700 1 -- 172.21.15.27:0/1178988490 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772400b820 con 0x7f7734142e90 2023-10-06T01:41:43.492 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.489+0000 7f77327fc700 1 -- 172.21.15.27:0/1178988490 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f772400b980 con 0x7f7734142e90 2023-10-06T01:41:43.492 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.489+0000 7f77327fc700 1 -- 172.21.15.27:0/1178988490 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772400bd20 con 0x7f7734142e90 2023-10-06T01:41:43.493 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.489+0000 7f7738c4a700 1 -- 172.21.15.27:0/1178988490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7734142e90 msgr2=0x7f7734069810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:43.493 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.489+0000 7f7738c4a700 1 --2- 172.21.15.27:0/1178988490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7734142e90 0x7f7734069810 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f773406fd60 tx=0x7f7724009250 comp rx=0 tx=0).stop 2023-10-06T01:41:43.493 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.489+0000 7f7738c4a700 1 -- 172.21.15.27:0/1178988490 shutdown_connections 2023-10-06T01:41:43.493 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.489+0000 7f7738c4a700 1 --2- 172.21.15.27:0/1178988490 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7734142e90 0x7f7734069810 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:43.493 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.489+0000 7f7738c4a700 1 --2- 172.21.15.27:0/1178988490 >> v2:172.21.15.124:3300/0 conn(0x7f7734069d50 0x7f773415d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:43.493 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.489+0000 7f7738c4a700 1 -- 172.21.15.27:0/1178988490 >> 172.21.15.27:0/1178988490 conn(0x7f77340ccfe0 msgr2=0x7f77340cd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:43.493 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.489+0000 7f7738c4a700 1 -- 172.21.15.27:0/1178988490 shutdown_connections 2023-10-06T01:41:43.493 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.489+0000 7f7738c4a700 1 -- 172.21.15.27:0/1178988490 wait complete. 2023-10-06T01:41:43.496 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.490+0000 7f7738c4a700 1 Processor -- start 2023-10-06T01:41:43.496 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.490+0000 7f7738c4a700 1 -- start start 2023-10-06T01:41:43.496 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.491+0000 7f7738c4a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7734069d50 0x7f7734169a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:43.496 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.491+0000 7f7738c4a700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7734142e90 0x7f77340cbae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:43.497 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.491+0000 7f7738c4a700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f773415de80 0x7f7734141950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:43.497 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.491+0000 7f7738c4a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77340cc8d0 con 0x7f7734069d50 2023-10-06T01:41:43.497 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.491+0000 7f7738c4a700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77340c6d90 con 0x7f7734142e90 2023-10-06T01:41:43.497 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.491+0000 7f7738c4a700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77340cc5f0 con 0x7f773415de80 2023-10-06T01:41:43.497 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.491+0000 7f77337fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7734069d50 0x7f7734169a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:43.497 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.491+0000 7f77337fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7734069d50 0x7f7734169a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34412/0 (socket says 172.21.15.27:34412) 2023-10-06T01:41:43.497 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.491+0000 7f77337fe700 1 -- 172.21.15.27:0/1205701377 learned_addr learned my addr 172.21.15.27:0/1205701377 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:43.497 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.491+0000 7f77337fe700 1 -- 172.21.15.27:0/1205701377 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f773415de80 msgr2=0x7f7734141950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:43.497 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.491+0000 7f7733fff700 1 --2- 172.21.15.27:0/1205701377 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f773415de80 0x7f7734141950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:43.498 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.491+0000 7f7732ffd700 1 --2- 172.21.15.27:0/1205701377 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7734142e90 0x7f77340cbae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:43.498 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.491+0000 7f77337fe700 1 --2- 172.21.15.27:0/1205701377 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f773415de80 0x7f7734141950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:43.498 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.492+0000 7f77337fe700 1 -- 172.21.15.27:0/1205701377 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7734142e90 msgr2=0x7f77340cbae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:43.498 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.492+0000 7f77337fe700 1 --2- 172.21.15.27:0/1205701377 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7734142e90 0x7f77340cbae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:43.498 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.492+0000 7f77337fe700 1 -- 172.21.15.27:0/1205701377 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77240094f0 con 0x7f7734069d50 2023-10-06T01:41:43.498 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.492+0000 7f77337fe700 1 --2- 172.21.15.27:0/1205701377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7734069d50 0x7f7734169a60 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7724009290 tx=0x7f7724016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:43.498 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.492+0000 7f771b7fe700 1 -- 172.21.15.27:0/1205701377 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7724011e20 con 0x7f7734069d50 2023-10-06T01:41:43.498 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.492+0000 7f771b7fe700 1 -- 172.21.15.27:0/1205701377 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f772401b680 con 0x7f7734069d50 2023-10-06T01:41:43.498 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.492+0000 7f7738c4a700 1 -- 172.21.15.27:0/1205701377 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77340c6740 con 0x7f7734069d50 2023-10-06T01:41:43.498 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.492+0000 7f771b7fe700 1 -- 172.21.15.27:0/1205701377 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772401ba40 con 0x7f7734069d50 2023-10-06T01:41:43.499 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.492+0000 7f7738c4a700 1 -- 172.21.15.27:0/1205701377 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77340c9230 con 0x7f7734069d50 2023-10-06T01:41:43.499 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.493+0000 7f7738c4a700 1 -- 172.21.15.27:0/1205701377 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f773404c3c0 con 0x7f7734069d50 2023-10-06T01:41:43.499 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.493+0000 7f771b7fe700 1 -- 172.21.15.27:0/1205701377 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f772401c450 con 0x7f7734069d50 2023-10-06T01:41:43.499 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.494+0000 7f771b7fe700 1 --2- 172.21.15.27:0/1205701377 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7720066f30 0x7f77200693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:43.499 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.494+0000 7f771b7fe700 1 -- 172.21.15.27:0/1205701377 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f77240863f0 con 0x7f7734069d50 2023-10-06T01:41:43.499 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.494+0000 7f7732ffd700 1 --2- 172.21.15.27:0/1205701377 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7720066f30 0x7f77200693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:43.499 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.494+0000 7f7732ffd700 1 --2- 172.21.15.27:0/1205701377 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7720066f30 0x7f77200693e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f77340b1ce0 tx=0x7f7728008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:43.500 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.498+0000 7f771b7fe700 1 -- 172.21.15.27:0/1205701377 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7724053460 con 0x7f7734069d50 2023-10-06T01:41:43.694 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.693+0000 7f7738c4a700 1 -- 172.21.15.27:0/1205701377 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7734069010 con 0x7f7734069d50 2023-10-06T01:41:43.694 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.693+0000 7f771b7fe700 1 -- 172.21.15.27:0/1205701377 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7724056a80 con 0x7f7734069d50 2023-10-06T01:41:43.695 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2023-10-06T01:41:43.697 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.695+0000 7f7738c4a700 1 -- 172.21.15.27:0/1205701377 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7720066f30 msgr2=0x7f77200693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:43.697 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.695+0000 7f7738c4a700 1 --2- 172.21.15.27:0/1205701377 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7720066f30 0x7f77200693e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f77340b1ce0 tx=0x7f7728008040 comp rx=0 tx=0).stop 2023-10-06T01:41:43.697 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.695+0000 7f7738c4a700 1 -- 172.21.15.27:0/1205701377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7734069d50 msgr2=0x7f7734169a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:43.697 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.695+0000 7f7738c4a700 1 --2- 172.21.15.27:0/1205701377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7734069d50 0x7f7734169a60 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7724009290 tx=0x7f7724016000 comp rx=0 tx=0).stop 2023-10-06T01:41:43.697 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.695+0000 7f7738c4a700 1 -- 172.21.15.27:0/1205701377 shutdown_connections 2023-10-06T01:41:43.698 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.695+0000 7f7738c4a700 1 --2- 172.21.15.27:0/1205701377 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7720066f30 0x7f77200693e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:43.698 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.695+0000 7f7738c4a700 1 --2- 172.21.15.27:0/1205701377 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f773415de80 0x7f7734141950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:43.698 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.695+0000 7f7738c4a700 1 --2- 172.21.15.27:0/1205701377 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7734069d50 0x7f7734169a60 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:43.698 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.696+0000 7f7738c4a700 1 --2- 172.21.15.27:0/1205701377 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7734142e90 0x7f77340cbae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:43.698 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.696+0000 7f7738c4a700 1 -- 172.21.15.27:0/1205701377 >> 172.21.15.27:0/1205701377 conn(0x7f77340ccfe0 msgr2=0x7f773413ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:43.698 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.696+0000 7f7738c4a700 1 -- 172.21.15.27:0/1205701377 shutdown_connections 2023-10-06T01:41:43.698 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.696+0000 7f7738c4a700 1 -- 172.21.15.27:0/1205701377 wait complete. 2023-10-06T01:41:43.709 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-06T01:41:43.831 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.829+0000 7fe2b8d49700 1 Processor -- start 2023-10-06T01:41:43.831 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.829+0000 7fe2b8d49700 1 -- start start 2023-10-06T01:41:43.833 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.830+0000 7fe2b8d49700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe2b406f270 0x7fe2b415d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:43.833 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.830+0000 7fe2b8d49700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2b40ae340 con 0x7fe2b406b470 2023-10-06T01:41:43.833 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.830+0000 7fe2b8d49700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2b40afaa0 con 0x7fe2b40c80d0 2023-10-06T01:41:43.834 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.830+0000 7fe2b8d49700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2b415e290 con 0x7fe2b406f270 2023-10-06T01:41:43.834 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.830+0000 7fe2b259c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe2b406b470 0x7fe2b40c79c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41052/0 (socket says 172.21.15.27:41052) 2023-10-06T01:41:43.834 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.830+0000 7fe2b259c700 1 -- 172.21.15.27:0/1951690237 learned_addr learned my addr 172.21.15.27:0/1951690237 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:43.834 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.831+0000 7fe2b2d9d700 1 --2- 172.21.15.27:0/1951690237 >> v2:172.21.15.27:3300/0 conn(0x7fe2b406f270 0x7fe2b415d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:43.834 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.831+0000 7fe2b159a700 1 -- 172.21.15.27:0/1951690237 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1860763554 0 0) 0x7fe2b40ae340 con 0x7fe2b406b470 2023-10-06T01:41:43.834 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.831+0000 7fe2b159a700 1 -- 172.21.15.27:0/1951690237 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2a0003570 con 0x7fe2b406b470 2023-10-06T01:41:43.834 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.831+0000 7fe2b2d9d700 1 -- 172.21.15.27:0/1951690237 >> v1:172.21.15.124:6789/0 conn(0x7fe2b40c80d0 legacy=0x7fe2b4159a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:43.834 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.831+0000 7fe2b2d9d700 1 -- 172.21.15.27:0/1951690237 >> v1:172.21.15.27:6789/0 conn(0x7fe2b406b470 legacy=0x7fe2b40c79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:43.835 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.831+0000 7fe2b2d9d700 1 -- 172.21.15.27:0/1951690237 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b406b840 con 0x7fe2b406f270 2023-10-06T01:41:43.835 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.831+0000 7fe2b2d9d700 1 --2- 172.21.15.27:0/1951690237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2b406f270 0x7fe2b415d2f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe2a40098e0 tx=0x7fe2a40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34e91de415af097a server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:43.835 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.831+0000 7fe2b159a700 1 -- 172.21.15.27:0/1951690237 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a400b640 con 0x7fe2b406f270 2023-10-06T01:41:43.835 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.831+0000 7fe2b159a700 1 -- 172.21.15.27:0/1951690237 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2a400b7a0 con 0x7fe2b406f270 2023-10-06T01:41:43.835 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.831+0000 7fe2b159a700 1 -- 172.21.15.27:0/1951690237 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a400bb40 con 0x7fe2b406f270 2023-10-06T01:41:43.835 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.832+0000 7fe2b8d49700 1 -- 172.21.15.27:0/1951690237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2b406f270 msgr2=0x7fe2b415d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:43.835 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.832+0000 7fe2b8d49700 1 --2- 172.21.15.27:0/1951690237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2b406f270 0x7fe2b415d2f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fe2a40098e0 tx=0x7fe2a40091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:43.835 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.832+0000 7fe2b8d49700 1 -- 172.21.15.27:0/1951690237 shutdown_connections 2023-10-06T01:41:43.835 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.832+0000 7fe2b8d49700 1 --2- 172.21.15.27:0/1951690237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2b406f270 0x7fe2b415d2f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:43.836 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.832+0000 7fe2b8d49700 1 -- 172.21.15.27:0/1951690237 >> 172.21.15.27:0/1951690237 conn(0x7fe2b406e760 msgr2=0x7fe2b406eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:43.836 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.832+0000 7fe2b8d49700 1 -- 172.21.15.27:0/1951690237 shutdown_connections 2023-10-06T01:41:43.836 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.832+0000 7fe2b8d49700 1 -- 172.21.15.27:0/1951690237 wait complete. 2023-10-06T01:41:43.840 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.833+0000 7fe2b8d49700 1 Processor -- start 2023-10-06T01:41:43.840 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.833+0000 7fe2b8d49700 1 -- start start 2023-10-06T01:41:43.840 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.834+0000 7fe2b8d49700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe2b406b470 0x7fe2b406a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:43.840 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.834+0000 7fe2b8d49700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe2b406f270 0x7fe2b4069cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:43.840 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.834+0000 7fe2b8d49700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2b40c80d0 0x7fe2b4157b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:43.840 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.834+0000 7fe2b8d49700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b406a230 con 0x7fe2b40c80d0 2023-10-06T01:41:43.840 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.834+0000 7fe2b8d49700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b4068900 con 0x7fe2b406f270 2023-10-06T01:41:43.841 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.834+0000 7fe2b8d49700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b406aa70 con 0x7fe2b406b470 2023-10-06T01:41:43.841 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.834+0000 7fe2b2d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2b40c80d0 0x7fe2b4157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:43.841 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.834+0000 7fe2b2d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2b40c80d0 0x7fe2b4157b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34432/0 (socket says 172.21.15.27:34432) 2023-10-06T01:41:43.841 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.834+0000 7fe2b2d9d700 1 -- 172.21.15.27:0/2126765170 learned_addr learned my addr 172.21.15.27:0/2126765170 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:43.842 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.834+0000 7fe2b259c700 1 --2- 172.21.15.27:0/2126765170 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe2b406b470 0x7fe2b406a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:43.842 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.834+0000 7fe2b1d9b700 1 --2- 172.21.15.27:0/2126765170 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe2b406f270 0x7fe2b4069cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:43.842 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.834+0000 7fe2b2d9d700 1 -- 172.21.15.27:0/2126765170 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe2b406b470 msgr2=0x7fe2b406a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:43.842 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.834+0000 7fe2b2d9d700 1 --2- 172.21.15.27:0/2126765170 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe2b406b470 0x7fe2b406a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:43.842 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.834+0000 7fe2b2d9d700 1 -- 172.21.15.27:0/2126765170 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe2b406f270 msgr2=0x7fe2b4069cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:43.842 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.834+0000 7fe2b2d9d700 1 --2- 172.21.15.27:0/2126765170 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe2b406f270 0x7fe2b4069cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:43.842 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.834+0000 7fe2b2d9d700 1 -- 172.21.15.27:0/2126765170 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a4009420 con 0x7fe2b40c80d0 2023-10-06T01:41:43.842 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.834+0000 7fe2b2d9d700 1 --2- 172.21.15.27:0/2126765170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2b40c80d0 0x7fe2b4157b90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe2a4006ef0 tx=0x7fe2a4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:43.842 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.834+0000 7fe2aa7fc700 1 -- 172.21.15.27:0/2126765170 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a400bb20 con 0x7fe2b40c80d0 2023-10-06T01:41:43.842 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.835+0000 7fe2aa7fc700 1 -- 172.21.15.27:0/2126765170 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2a4011420 con 0x7fe2b40c80d0 2023-10-06T01:41:43.842 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.835+0000 7fe2b8d49700 1 -- 172.21.15.27:0/2126765170 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2b41580d0 con 0x7fe2b40c80d0 2023-10-06T01:41:43.843 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.835+0000 7fe2aa7fc700 1 -- 172.21.15.27:0/2126765170 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a40117a0 con 0x7fe2b40c80d0 2023-10-06T01:41:43.843 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.835+0000 7fe2b8d49700 1 -- 172.21.15.27:0/2126765170 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2b40682b0 con 0x7fe2b40c80d0 2023-10-06T01:41:43.843 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.835+0000 7fe2b8d49700 1 -- 172.21.15.27:0/2126765170 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2b40ad480 con 0x7fe2b40c80d0 2023-10-06T01:41:43.843 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.836+0000 7fe2aa7fc700 1 -- 172.21.15.27:0/2126765170 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe2a4011b10 con 0x7fe2b40c80d0 2023-10-06T01:41:43.843 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.836+0000 7fe2aa7fc700 1 --2- 172.21.15.27:0/2126765170 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe294066f30 0x7fe2940693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:43.843 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.836+0000 7fe2aa7fc700 1 -- 172.21.15.27:0/2126765170 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fe2a4086170 con 0x7fe2b40c80d0 2023-10-06T01:41:43.843 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.837+0000 7fe2b259c700 1 --2- 172.21.15.27:0/2126765170 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe294066f30 0x7fe2940693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:43.843 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.837+0000 7fe2b259c700 1 --2- 172.21.15.27:0/2126765170 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe294066f30 0x7fe2940693e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac005c50 tx=0x7fe2ac009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:43.844 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:43.841+0000 7fe2aa7fc700 1 -- 172.21.15.27:0/2126765170 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe2a4053280 con 0x7fe2b40c80d0 2023-10-06T01:41:44.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.030+0000 7fe2b8d49700 1 -- 172.21.15.27:0/2126765170 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe2b4064050 con 0x7fe2b40c80d0 2023-10-06T01:41:44.032 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.030+0000 7fe2aa7fc700 1 -- 172.21.15.27:0/2126765170 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe2a40568a0 con 0x7fe2b40c80d0 2023-10-06T01:41:44.032 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2023-10-06T01:41:44.034 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.032+0000 7fe2b8d49700 1 -- 172.21.15.27:0/2126765170 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe294066f30 msgr2=0x7fe2940693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:44.035 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.032+0000 7fe2b8d49700 1 --2- 172.21.15.27:0/2126765170 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe294066f30 0x7fe2940693e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac005c50 tx=0x7fe2ac009040 comp rx=0 tx=0).stop 2023-10-06T01:41:44.035 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.032+0000 7fe2b8d49700 1 -- 172.21.15.27:0/2126765170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2b40c80d0 msgr2=0x7fe2b4157b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:44.035 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.032+0000 7fe2b8d49700 1 --2- 172.21.15.27:0/2126765170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2b40c80d0 0x7fe2b4157b90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe2a4006ef0 tx=0x7fe2a4017000 comp rx=0 tx=0).stop 2023-10-06T01:41:44.035 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.032+0000 7fe2b8d49700 1 -- 172.21.15.27:0/2126765170 shutdown_connections 2023-10-06T01:41:44.035 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.032+0000 7fe2b8d49700 1 --2- 172.21.15.27:0/2126765170 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe294066f30 0x7fe2940693e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:44.035 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.032+0000 7fe2b8d49700 1 --2- 172.21.15.27:0/2126765170 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe2b406b470 0x7fe2b406a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:44.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.032+0000 7fe2b8d49700 1 --2- 172.21.15.27:0/2126765170 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2b40c80d0 0x7fe2b4157b90 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:44.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.032+0000 7fe2b8d49700 1 --2- 172.21.15.27:0/2126765170 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe2b406f270 0x7fe2b4069cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:44.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.032+0000 7fe2b8d49700 1 -- 172.21.15.27:0/2126765170 >> 172.21.15.27:0/2126765170 conn(0x7fe2b406e760 msgr2=0x7fe2b41582a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:44.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.032+0000 7fe2b8d49700 1 -- 172.21.15.27:0/2126765170 shutdown_connections 2023-10-06T01:41:44.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.032+0000 7fe2b8d49700 1 -- 172.21.15.27:0/2126765170 wait complete. 2023-10-06T01:41:44.045 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-06T01:41:44.168 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.166+0000 7f83ecda0700 1 Processor -- start 2023-10-06T01:41:44.168 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.166+0000 7f83ecda0700 1 -- start start 2023-10-06T01:41:44.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.167+0000 7f83ecda0700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f83e8142e90 0x7f83e8069810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:44.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.167+0000 7f83ecda0700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83e8069380 con 0x7f83e8069d50 2023-10-06T01:41:44.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.167+0000 7f83ecda0700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83e80b63a0 con 0x7f83e815d9a0 2023-10-06T01:41:44.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.167+0000 7f83ecda0700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83e8068770 con 0x7f83e8142e90 2023-10-06T01:41:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.167+0000 7f83e6ffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f83e8069d50 0x7f83e815d290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41060/0 (socket says 172.21.15.27:41060) 2023-10-06T01:41:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.167+0000 7f83e6ffd700 1 -- 172.21.15.27:0/2723207768 learned_addr learned my addr 172.21.15.27:0/2723207768 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.167+0000 7f83e77fe700 1 --2- 172.21.15.27:0/2723207768 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f83e8142e90 0x7f83e8069810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.167+0000 7f83e67fc700 1 -- 172.21.15.27:0/2723207768 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3659928269 0 0) 0x7f83e8069380 con 0x7f83e8069d50 2023-10-06T01:41:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.167+0000 7f83e67fc700 1 -- 172.21.15.27:0/2723207768 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83d0003570 con 0x7f83e8069d50 2023-10-06T01:41:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.167+0000 7f83e67fc700 1 -- 172.21.15.27:0/2723207768 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 975173835 0 0) 0x7f83e80b63a0 con 0x7f83e815d9a0 2023-10-06T01:41:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.167+0000 7f83e67fc700 1 -- 172.21.15.27:0/2723207768 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83e8069380 con 0x7f83e815d9a0 2023-10-06T01:41:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.168+0000 7f83e77fe700 1 -- 172.21.15.27:0/2723207768 >> v1:172.21.15.124:6789/0 conn(0x7f83e815d9a0 legacy=0x7f83e80b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.168+0000 7f83e77fe700 1 -- 172.21.15.27:0/2723207768 >> v1:172.21.15.27:6789/0 conn(0x7f83e8069d50 legacy=0x7f83e815d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.168+0000 7f83e77fe700 1 -- 172.21.15.27:0/2723207768 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e8068a50 con 0x7f83e8142e90 2023-10-06T01:41:44.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.168+0000 7f83e77fe700 1 --2- 172.21.15.27:0/2723207768 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f83e8142e90 0x7f83e8069810 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f83d8009910 tx=0x7f83d80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f846e071895ac61d server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.168+0000 7f83e67fc700 1 -- 172.21.15.27:0/2723207768 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d8011630 con 0x7f83e8142e90 2023-10-06T01:41:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.168+0000 7f83e67fc700 1 -- 172.21.15.27:0/2723207768 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d8011790 con 0x7f83e8142e90 2023-10-06T01:41:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.168+0000 7f83e67fc700 1 -- 172.21.15.27:0/2723207768 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d8011b30 con 0x7f83e8142e90 2023-10-06T01:41:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.168+0000 7f83ecda0700 1 -- 172.21.15.27:0/2723207768 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f83e8142e90 msgr2=0x7f83e8069810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.168+0000 7f83ecda0700 1 --2- 172.21.15.27:0/2723207768 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f83e8142e90 0x7f83e8069810 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f83d8009910 tx=0x7f83d80091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.168+0000 7f83ecda0700 1 -- 172.21.15.27:0/2723207768 shutdown_connections 2023-10-06T01:41:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.168+0000 7f83ecda0700 1 --2- 172.21.15.27:0/2723207768 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f83e8142e90 0x7f83e8069810 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.168+0000 7f83ecda0700 1 -- 172.21.15.27:0/2723207768 >> 172.21.15.27:0/2723207768 conn(0x7f83e80ccfe0 msgr2=0x7f83e80cd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.169+0000 7f83ecda0700 1 -- 172.21.15.27:0/2723207768 shutdown_connections 2023-10-06T01:41:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.169+0000 7f83ecda0700 1 -- 172.21.15.27:0/2723207768 wait complete. 2023-10-06T01:41:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.170+0000 7f83ecda0700 1 Processor -- start 2023-10-06T01:41:44.173 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.170+0000 7f83ecda0700 1 -- start start 2023-10-06T01:41:44.176 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.171+0000 7f83ecda0700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f83e8069d50 0x7f83e80c8770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:44.176 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.171+0000 7f83ecda0700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83e8142e90 0x7f83e816d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:44.176 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.171+0000 7f83ecda0700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f83e815d9a0 0x7f83e816d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:44.176 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.171+0000 7f83ecda0700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e80c9bd0 con 0x7f83e8142e90 2023-10-06T01:41:44.176 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.171+0000 7f83ecda0700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e80c8fc0 con 0x7f83e8069d50 2023-10-06T01:41:44.176 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.171+0000 7f83ecda0700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83e80c92a0 con 0x7f83e815d9a0 2023-10-06T01:41:44.176 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.171+0000 7f83e6ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83e8142e90 0x7f83e816d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:44.176 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.171+0000 7f83e6ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83e8142e90 0x7f83e816d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34444/0 (socket says 172.21.15.27:34444) 2023-10-06T01:41:44.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.171+0000 7f83e6ffd700 1 -- 172.21.15.27:0/3076155766 learned_addr learned my addr 172.21.15.27:0/3076155766 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:44.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.171+0000 7f83e77fe700 1 --2- 172.21.15.27:0/3076155766 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f83e8069d50 0x7f83e80c8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:44.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.171+0000 7f83e7fff700 1 --2- 172.21.15.27:0/3076155766 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f83e815d9a0 0x7f83e816d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:44.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.171+0000 7f83e6ffd700 1 -- 172.21.15.27:0/3076155766 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f83e815d9a0 msgr2=0x7f83e816d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:44.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.171+0000 7f83e6ffd700 1 --2- 172.21.15.27:0/3076155766 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f83e815d9a0 0x7f83e816d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:44.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.171+0000 7f83e6ffd700 1 -- 172.21.15.27:0/3076155766 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f83e8069d50 msgr2=0x7f83e80c8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:44.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.171+0000 7f83e6ffd700 1 --2- 172.21.15.27:0/3076155766 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f83e8069d50 0x7f83e80c8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:44.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.171+0000 7f83e6ffd700 1 -- 172.21.15.27:0/3076155766 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d8009420 con 0x7f83e8142e90 2023-10-06T01:41:44.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.171+0000 7f83e6ffd700 1 --2- 172.21.15.27:0/3076155766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83e8142e90 0x7f83e816d3d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f83dc00c790 tx=0x7f83dc007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:44.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.171+0000 7f83cf7fe700 1 -- 172.21.15.27:0/3076155766 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83dc005b30 con 0x7f83e8142e90 2023-10-06T01:41:44.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.171+0000 7f83cf7fe700 1 -- 172.21.15.27:0/3076155766 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83dc005c90 con 0x7f83e8142e90 2023-10-06T01:41:44.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.171+0000 7f83cf7fe700 1 -- 172.21.15.27:0/3076155766 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83dc007300 con 0x7f83e8142e90 2023-10-06T01:41:44.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.171+0000 7f83ecda0700 1 -- 172.21.15.27:0/3076155766 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83e80c9580 con 0x7f83e8142e90 2023-10-06T01:41:44.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.171+0000 7f83ecda0700 1 -- 172.21.15.27:0/3076155766 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83e80c8cb0 con 0x7f83e8142e90 2023-10-06T01:41:44.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.172+0000 7f83ecda0700 1 -- 172.21.15.27:0/3076155766 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83e80c9d40 con 0x7f83e8142e90 2023-10-06T01:41:44.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.173+0000 7f83cf7fe700 1 -- 172.21.15.27:0/3076155766 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f83dc01b3b0 con 0x7f83e8142e90 2023-10-06T01:41:44.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.173+0000 7f83cf7fe700 1 --2- 172.21.15.27:0/3076155766 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f83d4066f30 0x7f83d40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:44.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.173+0000 7f83cf7fe700 1 -- 172.21.15.27:0/3076155766 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f83dc083be0 con 0x7f83e8142e90 2023-10-06T01:41:44.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.173+0000 7f83e77fe700 1 --2- 172.21.15.27:0/3076155766 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f83d4066f30 0x7f83d40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:44.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.174+0000 7f83e77fe700 1 --2- 172.21.15.27:0/3076155766 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f83d4066f30 0x7f83d40693e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f83d8009860 tx=0x7f83d8017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:44.179 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.177+0000 7f83cf7fe700 1 -- 172.21.15.27:0/3076155766 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f83dc050cf0 con 0x7f83e8142e90 2023-10-06T01:41:44.372 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.370+0000 7f83ecda0700 1 -- 172.21.15.27:0/3076155766 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f83e804c3c0 con 0x7f83e8142e90 2023-10-06T01:41:44.372 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.371+0000 7f83cf7fe700 1 -- 172.21.15.27:0/3076155766 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f83dc017020 con 0x7f83e8142e90 2023-10-06T01:41:44.372 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2023-10-06T01:41:44.374 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.373+0000 7f83ecda0700 1 -- 172.21.15.27:0/3076155766 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f83d4066f30 msgr2=0x7f83d40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:44.375 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.373+0000 7f83ecda0700 1 --2- 172.21.15.27:0/3076155766 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f83d4066f30 0x7f83d40693e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f83d8009860 tx=0x7f83d8017000 comp rx=0 tx=0).stop 2023-10-06T01:41:44.375 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.373+0000 7f83ecda0700 1 -- 172.21.15.27:0/3076155766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83e8142e90 msgr2=0x7f83e816d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:44.375 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.373+0000 7f83ecda0700 1 --2- 172.21.15.27:0/3076155766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83e8142e90 0x7f83e816d3d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f83dc00c790 tx=0x7f83dc007ea0 comp rx=0 tx=0).stop 2023-10-06T01:41:44.375 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.373+0000 7f83ecda0700 1 -- 172.21.15.27:0/3076155766 shutdown_connections 2023-10-06T01:41:44.375 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.373+0000 7f83ecda0700 1 --2- 172.21.15.27:0/3076155766 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f83d4066f30 0x7f83d40693e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:44.375 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.373+0000 7f83ecda0700 1 --2- 172.21.15.27:0/3076155766 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f83e815d9a0 0x7f83e816d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:44.375 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.373+0000 7f83ecda0700 1 --2- 172.21.15.27:0/3076155766 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f83e8069d50 0x7f83e80c8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:44.375 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.373+0000 7f83ecda0700 1 --2- 172.21.15.27:0/3076155766 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f83e8142e90 0x7f83e816d3d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:44.375 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.373+0000 7f83ecda0700 1 -- 172.21.15.27:0/3076155766 >> 172.21.15.27:0/3076155766 conn(0x7f83e80ccfe0 msgr2=0x7f83e80b5db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:44.375 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.373+0000 7f83ecda0700 1 -- 172.21.15.27:0/3076155766 shutdown_connections 2023-10-06T01:41:44.375 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.373+0000 7f83ecda0700 1 -- 172.21.15.27:0/3076155766 wait complete. 2023-10-06T01:41:44.386 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T01:41:44.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.505+0000 7fdb686de700 1 Processor -- start 2023-10-06T01:41:44.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.505+0000 7fdb686de700 1 -- start start 2023-10-06T01:41:44.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.506+0000 7fdb686de700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdb601571d0 0x7fdb6015a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:44.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.506+0000 7fdb686de700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fdb600c79b0 0x7fdb6006c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:44.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.506+0000 7fdb686de700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb600cce40 con 0x7fdb6006c6d0 2023-10-06T01:41:44.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.506+0000 7fdb686de700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb6006cb70 con 0x7fdb600c79b0 2023-10-06T01:41:44.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.506+0000 7fdb686de700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6015ac80 con 0x7fdb601571d0 2023-10-06T01:41:44.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.506+0000 7fdb66c7b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdb6006c6d0 0x7fdb6015d310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41070/0 (socket says 172.21.15.27:41070) 2023-10-06T01:41:44.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.506+0000 7fdb66c7b700 1 -- 172.21.15.27:0/63386103 learned_addr learned my addr 172.21.15.27:0/63386103 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:44.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.506+0000 7fdb6647a700 1 --2- 172.21.15.27:0/63386103 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdb601571d0 0x7fdb6015a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:44.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.506+0000 7fdb65c79700 1 --2- 172.21.15.27:0/63386103 >> v2:172.21.15.124:3300/0 conn(0x7fdb600c79b0 0x7fdb6006c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:44.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.506+0000 7fdb65478700 1 -- 172.21.15.27:0/63386103 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1539863705 0 0) 0x7fdb600cce40 con 0x7fdb6006c6d0 2023-10-06T01:41:44.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.507+0000 7fdb65478700 1 -- 172.21.15.27:0/63386103 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb48003570 con 0x7fdb6006c6d0 2023-10-06T01:41:44.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.507+0000 7fdb65478700 1 -- 172.21.15.27:0/63386103 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3383149308 0 0) 0x7fdb48003570 con 0x7fdb6006c6d0 2023-10-06T01:41:44.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.507+0000 7fdb65478700 1 -- 172.21.15.27:0/63386103 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdb601571d0 msgr2=0x7fdb6015a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:44.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.507+0000 7fdb65478700 1 --2- 172.21.15.27:0/63386103 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdb601571d0 0x7fdb6015a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:44.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.507+0000 7fdb65478700 1 -- 172.21.15.27:0/63386103 >> v2:172.21.15.124:3300/0 conn(0x7fdb600c79b0 msgr2=0x7fdb6006c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:44.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.507+0000 7fdb65478700 1 --2- 172.21.15.27:0/63386103 >> v2:172.21.15.124:3300/0 conn(0x7fdb600c79b0 0x7fdb6006c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:44.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.507+0000 7fdb65478700 1 -- 172.21.15.27:0/63386103 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb600cbf50 con 0x7fdb6006c6d0 2023-10-06T01:41:44.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.507+0000 7fdb65478700 1 -- 172.21.15.27:0/63386103 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2619078944 0 0) 0x7fdb500017f0 con 0x7fdb6006c6d0 2023-10-06T01:41:44.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.507+0000 7fdb65478700 1 -- 172.21.15.27:0/63386103 >> v1:172.21.15.27:6789/0 conn(0x7fdb6006c6d0 legacy=0x7fdb6015d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:44.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.507+0000 7fdb65478700 1 --2- 172.21.15.27:0/63386103 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdb48003a60 0x7fdb48003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:44.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.507+0000 7fdb65478700 1 --2- 172.21.15.27:0/63386103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb48004810 0x7fdb48004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:44.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.507+0000 7fdb65478700 1 --2- 172.21.15.27:0/63386103 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdb601571d0 0x7fdb480051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:44.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.507+0000 7fdb65478700 1 -- 172.21.15.27:0/63386103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6006cb70 con 0x7fdb48004810 2023-10-06T01:41:44.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.507+0000 7fdb65478700 1 -- 172.21.15.27:0/63386103 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6015ac80 con 0x7fdb48003a60 2023-10-06T01:41:44.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.507+0000 7fdb65478700 1 -- 172.21.15.27:0/63386103 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4800ba60 con 0x7fdb601571d0 2023-10-06T01:41:44.513 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.507+0000 7fdb65c79700 1 --2- 172.21.15.27:0/63386103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb48004810 0x7fdb48004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:44.513 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.507+0000 7fdb6647a700 1 --2- 172.21.15.27:0/63386103 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdb48003a60 0x7fdb48003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:44.513 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.507+0000 7fdb66c7b700 1 --2- 172.21.15.27:0/63386103 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdb601571d0 0x7fdb480051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:44.513 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.507+0000 7fdb65c79700 1 -- 172.21.15.27:0/63386103 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdb601571d0 msgr2=0x7fdb480051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:44.513 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.507+0000 7fdb65c79700 1 --2- 172.21.15.27:0/63386103 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdb601571d0 0x7fdb480051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:44.513 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.507+0000 7fdb65c79700 1 -- 172.21.15.27:0/63386103 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdb48003a60 msgr2=0x7fdb48003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:44.513 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.507+0000 7fdb65c79700 1 --2- 172.21.15.27:0/63386103 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdb48003a60 0x7fdb48003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:44.513 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.507+0000 7fdb65c79700 1 -- 172.21.15.27:0/63386103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb48003570 con 0x7fdb48004810 2023-10-06T01:41:44.513 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.508+0000 7fdb65c79700 1 --2- 172.21.15.27:0/63386103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb48004810 0x7fdb48004c80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fdb54009c30 tx=0x7fdb540091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c3de12c341f37d server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:44.513 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.508+0000 7fdb65478700 1 -- 172.21.15.27:0/63386103 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb5400e710 con 0x7fdb48004810 2023-10-06T01:41:44.513 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.508+0000 7fdb65478700 1 -- 172.21.15.27:0/63386103 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb5400e870 con 0x7fdb48004810 2023-10-06T01:41:44.513 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.508+0000 7fdb65478700 1 -- 172.21.15.27:0/63386103 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb5400ec30 con 0x7fdb48004810 2023-10-06T01:41:44.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.508+0000 7fdb686de700 1 -- 172.21.15.27:0/63386103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb48004810 msgr2=0x7fdb48004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:44.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.508+0000 7fdb686de700 1 --2- 172.21.15.27:0/63386103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb48004810 0x7fdb48004c80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fdb54009c30 tx=0x7fdb540091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:44.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.508+0000 7fdb66c7b700 1 -- 172.21.15.27:0/63386103 reap_dead start 2023-10-06T01:41:44.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.508+0000 7fdb686de700 1 -- 172.21.15.27:0/63386103 shutdown_connections 2023-10-06T01:41:44.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.508+0000 7fdb686de700 1 -- 172.21.15.27:0/63386103 >> 172.21.15.27:0/63386103 conn(0x7fdb6006e6a0 msgr2=0x7fdb6006eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:44.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.508+0000 7fdb686de700 1 -- 172.21.15.27:0/63386103 shutdown_connections 2023-10-06T01:41:44.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.508+0000 7fdb686de700 1 -- 172.21.15.27:0/63386103 wait complete. 2023-10-06T01:41:44.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.509+0000 7fdb686de700 1 Processor -- start 2023-10-06T01:41:44.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.509+0000 7fdb686de700 1 -- start start 2023-10-06T01:41:44.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.510+0000 7fdb686de700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb48004810 0x7fdb6015bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:44.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.510+0000 7fdb686de700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdb60153530 0x7fdb600c6900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:44.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.510+0000 7fdb686de700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdb6014f6d0 0x7fdb6015d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:44.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.510+0000 7fdb686de700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb601539a0 con 0x7fdb48004810 2023-10-06T01:41:44.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.510+0000 7fdb686de700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6015c2b0 con 0x7fdb6014f6d0 2023-10-06T01:41:44.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.510+0000 7fdb686de700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6015c590 con 0x7fdb60153530 2023-10-06T01:41:44.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.510+0000 7fdb6647a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb48004810 0x7fdb6015bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:44.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.510+0000 7fdb6647a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb48004810 0x7fdb6015bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34462/0 (socket says 172.21.15.27:34462) 2023-10-06T01:41:44.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.510+0000 7fdb6647a700 1 -- 172.21.15.27:0/1706986776 learned_addr learned my addr 172.21.15.27:0/1706986776 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:44.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.510+0000 7fdb65c79700 1 --2- 172.21.15.27:0/1706986776 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdb60153530 0x7fdb600c6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:44.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.510+0000 7fdb66c7b700 1 --2- 172.21.15.27:0/1706986776 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdb6014f6d0 0x7fdb6015d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:44.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.510+0000 7fdb6647a700 1 -- 172.21.15.27:0/1706986776 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdb60153530 msgr2=0x7fdb600c6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:44.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.510+0000 7fdb6647a700 1 --2- 172.21.15.27:0/1706986776 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdb60153530 0x7fdb600c6900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:44.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.510+0000 7fdb6647a700 1 -- 172.21.15.27:0/1706986776 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdb6014f6d0 msgr2=0x7fdb6015d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:44.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.510+0000 7fdb6647a700 1 --2- 172.21.15.27:0/1706986776 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdb6014f6d0 0x7fdb6015d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:44.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.510+0000 7fdb6647a700 1 -- 172.21.15.27:0/1706986776 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb6015c870 con 0x7fdb48004810 2023-10-06T01:41:44.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.510+0000 7fdb65c79700 1 --2- 172.21.15.27:0/1706986776 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdb60153530 0x7fdb600c6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:44.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.510+0000 7fdb66c7b700 1 --2- 172.21.15.27:0/1706986776 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdb6014f6d0 0x7fdb6015d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:44.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.511+0000 7fdb6647a700 1 --2- 172.21.15.27:0/1706986776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb48004810 0x7fdb6015bc00 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fdb58011d40 tx=0x7fdb580091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:44.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.511+0000 7fdb5e7fc700 1 -- 172.21.15.27:0/1706986776 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb5800fac0 con 0x7fdb48004810 2023-10-06T01:41:44.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.511+0000 7fdb5e7fc700 1 -- 172.21.15.27:0/1706986776 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb5800fc20 con 0x7fdb48004810 2023-10-06T01:41:44.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.511+0000 7fdb686de700 1 -- 172.21.15.27:0/1706986776 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb6015cb50 con 0x7fdb48004810 2023-10-06T01:41:44.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.511+0000 7fdb5e7fc700 1 -- 172.21.15.27:0/1706986776 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb58014440 con 0x7fdb48004810 2023-10-06T01:41:44.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.511+0000 7fdb686de700 1 -- 172.21.15.27:0/1706986776 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb600c6e40 con 0x7fdb48004810 2023-10-06T01:41:44.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.512+0000 7fdb686de700 1 -- 172.21.15.27:0/1706986776 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb60064050 con 0x7fdb48004810 2023-10-06T01:41:44.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.512+0000 7fdb5e7fc700 1 -- 172.21.15.27:0/1706986776 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fdb58014870 con 0x7fdb48004810 2023-10-06T01:41:44.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.512+0000 7fdb5e7fc700 1 --2- 172.21.15.27:0/1706986776 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdb4c066f30 0x7fdb4c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:44.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.512+0000 7fdb5e7fc700 1 -- 172.21.15.27:0/1706986776 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fdb58018030 con 0x7fdb48004810 2023-10-06T01:41:44.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.513+0000 7fdb65c79700 1 --2- 172.21.15.27:0/1706986776 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdb4c066f30 0x7fdb4c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:44.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.513+0000 7fdb65c79700 1 --2- 172.21.15.27:0/1706986776 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdb4c066f30 0x7fdb4c0693e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fdb540091e0 tx=0x7fdb5401a000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:44.520 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.517+0000 7fdb5e7fc700 1 -- 172.21.15.27:0/1706986776 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdb58052280 con 0x7fdb48004810 2023-10-06T01:41:44.711 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.709+0000 7fdb686de700 1 -- 172.21.15.27:0/1706986776 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb6015cec0 con 0x7fdb48004810 2023-10-06T01:41:44.711 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.710+0000 7fdb5e7fc700 1 -- 172.21.15.27:0/1706986776 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16866 (secure 0 0 0) 0x7fdb580558a0 con 0x7fdb48004810 2023-10-06T01:41:44.712 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:44.712 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":20,"fsid":"20a7bff0-74f8-4f2f-b6c6-899e804bc965","created":"2023-10-06T01:40:54.474215+0000","modified":"2023-10-06T01:41:14.324546+0000","last_up_change":"2023-10-06T01:41:05.242847+0000","last_in_change":"2023-10-06T01:40:56.890960+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-10-06T01:41:05.685386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-10-06T01:41:06.092995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T01:41:10.434257+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T01:41:11.700262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b902c86d-800c-40c5-b152-b545e50d02c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":90644},{"type":"v1","addr":"172.21.15.27:6801","nonce":90644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":90644},{"type":"v1","addr":"172.21.15.27:6803","nonce":90644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":90644},{"type":"v1","addr":"172.21.15.27:6807","nonce":90644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":90644},{"type":"v1","addr":"172.21.15.27:6805","nonce":90644}]},"public_addr":"172.21.15.27:6801/90644","cluster_addr":"172.21.15.27:6803/90644","heartbeat_back_addr":"172.21.15.27:6807/90644","heartbeat_front_addr":"172.21.15.27:6805/90644","state":["exists","up"]},{"osd":1,"uuid":"5b4a70e5-b06e-48cd-bfc4-7962c5d05386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":90643},{"type":"v1","addr":"172.21.15.27:6809","nonce":90643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":90643},{"type":"v1","addr":"172.21.15.27:6811","nonce":90643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":90643},{"type":"v1","addr":"172.21.15.27:6815","nonce":90643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":90643},{"type":"v1","addr":"172.21.15.27:6813","nonce":90643}]},"public_addr":"172.21.15.27:6809/90643","cluster_addr":"172.21.15.27:6811/90643","heartbeat_back_addr":"172.21.15.27:6815/90643","heartbeat_front_addr":"172.21.15.27:6813/90643","state":["exists","up"]},{"osd":2,"uuid":"22d9de46-4073-4336-9122-5227e601eca0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":90645},{"type":"v1","addr":"172.21.15.27:6817","nonce":90645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":90645},{"type":"v1","addr":"172.21.15.27:6819","nonce":90645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":90645},{"type":"v1","addr":"172.21.15.27:6823","nonce":90645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":90645},{"type":"v1","addr":"172.21.15.27:6821","nonce":90645}]},"public_addr":"172.21.15.27:6817/90645","cluster_addr":"172.21.15.27:6819/90645","heartbeat_back_addr":"172.21.15.27:6823/90645","heartbeat_front_addr":"172.21.15.27:6821/90645","state":["exists","up"]},{"osd":3,"uuid":"438bb2d3-34e2-4b7c-9734-8360c1b67650","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":90668},{"type":"v1","addr":"172.21.15.27:6825","nonce":90668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":90668},{"type":"v1","addr":"172.21.15.27:6827","nonce":90668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":90668},{"type":"v1","addr":"172.21.15.27:6831","nonce":90668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":90668},{"type":"v1","addr":"172.21.15.27:6829","nonce":90668}]},"public_addr":"172.21.15.27:6825/90668","cluster_addr":"172.21.15.27:6827/90668","heartbeat_back_addr":"172.21.15.27:6831/90668","heartbeat_front_addr":"172.21.15.27:6829/90668","state":["exists","up"]},{"osd":4,"uuid":"e1956a8e-e0f8-4e7c-acbb-40037415a354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":90863},{"type":"v1","addr":"172.21.15.124:6801","nonce":90863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":90863},{"type":"v1","addr":"172.21.15.124:6803","nonce":90863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":90863},{"type":"v1","addr":"172.21.15.124:6807","nonce":90863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":90863},{"type":"v1","addr":"172.21.15.124:6805","nonce":90863}]},"public_addr":"172.21.15.124:6801/90863","cluster_addr":"172.21.15.124:6803/90863","heartbeat_back_addr":"172.21.15.124:6807/90863","heartbeat_front_addr":"172.21.15.124:6805/90863","state":["exists","up"]},{"osd":5,"uuid":"306c59d4-846d-4b73-b1c9-051bec9e9368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":90885},{"type":"v1","addr":"172.21.15.124:6817","nonce":90885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":90885},{"type":"v1","addr":"172.21.15.124:6819","nonce":90885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":90885},{"type":"v1","addr":"172.21.15.124:6823","nonce":90885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":90885},{"type":"v1","addr":"172.21.15.124:6821","nonce":90885}]},"public_addr":"172.21.15.124:6817/90885","cluster_addr":"172.21.15.124:6819/90885","heartbeat_back_addr":"172.21.15.124:6823/90885","heartbeat_front_addr":"172.21.15.124:6821/90885","state":["exists","up"]},{"osd":6,"uuid":"15463e53-ac8d-47d1-b4f6-c0d31f29e4a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":90864},{"type":"v1","addr":"172.21.15.124:6809","nonce":90864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":90864},{"type":"v1","addr":"172.21.15.124:6811","nonce":90864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":90864},{"type":"v1","addr":"172.21.15.124:6815","nonce":90864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":90864},{"type":"v1","addr":"172.21.15.124:6813","nonce":90864}]},"public_addr":"172.21.15.124:6809/90864","cluster_addr":"172.21.15.124:6811/90864","heartbeat_back_addr":"172.21.15.124:6815/90864","heartbeat_front_addr":"172.21.15.124:6813/90864","state":["exists","up"]},{"osd":7,"uuid":"ae463a90-a9d3-498f-86fa-47a35b3fe889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":90894},{"type":"v1","addr":"172.21.15.124:6825","nonce":90894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":90894},{"type":"v1","addr":"172.21.15.124:6827","nonce":90894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":90894},{"type":"v1","addr":"172.21.15.124:6831","nonce":90894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":90894},{"type":"v1","addr":"172.21.15.124:6829","nonce":90894}]},"public_addr":"172.21.15.124:6825/90894","cluster_addr":"172.21.15.124:6827/90894","heartbeat_back_addr":"172.21.15.124:6831/90894","heartbeat_front_addr":"172.21.15.124:6829/90894","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.827704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.839951+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.032230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.050605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.987908+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.031290+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.080707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.053692+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T01:41:44.714 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.712+0000 7fdb686de700 1 -- 172.21.15.27:0/1706986776 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdb4c066f30 msgr2=0x7fdb4c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:44.714 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.713+0000 7fdb686de700 1 --2- 172.21.15.27:0/1706986776 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdb4c066f30 0x7fdb4c0693e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fdb540091e0 tx=0x7fdb5401a000 comp rx=0 tx=0).stop 2023-10-06T01:41:44.714 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.713+0000 7fdb686de700 1 -- 172.21.15.27:0/1706986776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb48004810 msgr2=0x7fdb6015bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:44.715 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.713+0000 7fdb686de700 1 --2- 172.21.15.27:0/1706986776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb48004810 0x7fdb6015bc00 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fdb58011d40 tx=0x7fdb580091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:44.715 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.713+0000 7fdb686de700 1 -- 172.21.15.27:0/1706986776 shutdown_connections 2023-10-06T01:41:44.715 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.713+0000 7fdb686de700 1 --2- 172.21.15.27:0/1706986776 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdb4c066f30 0x7fdb4c0693e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:44.715 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.713+0000 7fdb686de700 1 --2- 172.21.15.27:0/1706986776 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdb6014f6d0 0x7fdb6015d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:44.715 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.713+0000 7fdb686de700 1 --2- 172.21.15.27:0/1706986776 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdb48004810 0x7fdb6015bc00 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:44.715 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.713+0000 7fdb686de700 1 --2- 172.21.15.27:0/1706986776 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdb60153530 0x7fdb600c6900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:44.715 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.713+0000 7fdb686de700 1 -- 172.21.15.27:0/1706986776 >> 172.21.15.27:0/1706986776 conn(0x7fdb6006e6a0 msgr2=0x7fdb600caf20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:44.715 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.713+0000 7fdb686de700 1 -- 172.21.15.27:0/1706986776 shutdown_connections 2023-10-06T01:41:44.715 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.713+0000 7fdb686de700 1 -- 172.21.15.27:0/1706986776 wait complete. 2023-10-06T01:41:44.726 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-10-06T01:41:05.685386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-10-06T01:41:06.092995+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-06T01:41:10.434257+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-10-06T01:41:11.700262+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-06T01:41:44.727 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-06T01:41:44.849 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.848+0000 7fdfe62ff700 1 Processor -- start 2023-10-06T01:41:44.849 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.848+0000 7fdfe62ff700 1 -- start start 2023-10-06T01:41:44.851 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.848+0000 7fdfe62ff700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdfe0159c50 0x7fdfe015e180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:44.851 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.848+0000 7fdfe62ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fdfe00694f0 0x7fdfe00aa970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:44.852 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.848+0000 7fdfe62ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfe00ab0c0 con 0x7fdfe0069120 2023-10-06T01:41:44.852 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.848+0000 7fdfe62ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfe00aaeb0 con 0x7fdfe00694f0 2023-10-06T01:41:44.852 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.848+0000 7fdfe62ff700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe00ab3a0 con 0x7fdfe0159c50 2023-10-06T01:41:44.852 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.849+0000 7fdfdffff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdfe0069120 0x7fdfe0159540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:41080/0 (socket says 172.21.15.27:41080) 2023-10-06T01:41:44.852 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.849+0000 7fdfdffff700 1 -- 172.21.15.27:0/2960046768 learned_addr learned my addr 172.21.15.27:0/2960046768 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:44.852 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.849+0000 7fdfe489c700 1 --2- 172.21.15.27:0/2960046768 >> v2:172.21.15.27:3300/0 conn(0x7fdfe00694f0 0x7fdfe00aa970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:44.852 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.849+0000 7fdfdf7fe700 1 --2- 172.21.15.27:0/2960046768 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdfe0159c50 0x7fdfe015e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:44.852 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.849+0000 7fdfdeffd700 1 -- 172.21.15.27:0/2960046768 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4033972646 0 0) 0x7fdfe00ab0c0 con 0x7fdfe0069120 2023-10-06T01:41:44.852 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.849+0000 7fdfdeffd700 1 -- 172.21.15.27:0/2960046768 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfcc003570 con 0x7fdfe0069120 2023-10-06T01:41:44.852 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.849+0000 7fdfe489c700 1 -- 172.21.15.27:0/2960046768 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdfe0159c50 msgr2=0x7fdfe015e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:44.852 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.849+0000 7fdfe489c700 1 --2- 172.21.15.27:0/2960046768 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdfe0159c50 0x7fdfe015e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:44.852 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.849+0000 7fdfe489c700 1 -- 172.21.15.27:0/2960046768 >> v1:172.21.15.27:6789/0 conn(0x7fdfe0069120 legacy=0x7fdfe0159540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:44.853 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.849+0000 7fdfe489c700 1 -- 172.21.15.27:0/2960046768 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfe00ab680 con 0x7fdfe00694f0 2023-10-06T01:41:44.853 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.849+0000 7fdfdf7fe700 1 --2- 172.21.15.27:0/2960046768 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdfe0159c50 0x7fdfe015e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:44.853 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.849+0000 7fdfe489c700 1 --2- 172.21.15.27:0/2960046768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfe00694f0 0x7fdfe00aa970 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8009e90 tx=0x7fdfd8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5843588cf9c920e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:44.853 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.849+0000 7fdfdeffd700 1 -- 172.21.15.27:0/2960046768 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd800e070 con 0x7fdfe00694f0 2023-10-06T01:41:44.853 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.849+0000 7fdfdeffd700 1 -- 172.21.15.27:0/2960046768 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfd800bbf0 con 0x7fdfe00694f0 2023-10-06T01:41:44.853 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.849+0000 7fdfdeffd700 1 -- 172.21.15.27:0/2960046768 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd8012430 con 0x7fdfe00694f0 2023-10-06T01:41:44.853 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.850+0000 7fdfe62ff700 1 -- 172.21.15.27:0/2960046768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfe00694f0 msgr2=0x7fdfe00aa970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:44.853 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.850+0000 7fdfe62ff700 1 --2- 172.21.15.27:0/2960046768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfe00694f0 0x7fdfe00aa970 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fdfd8009e90 tx=0x7fdfd8009780 comp rx=0 tx=0).stop 2023-10-06T01:41:44.853 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.850+0000 7fdfe62ff700 1 -- 172.21.15.27:0/2960046768 shutdown_connections 2023-10-06T01:41:44.853 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.850+0000 7fdfe62ff700 1 --2- 172.21.15.27:0/2960046768 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfe00694f0 0x7fdfe00aa970 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:44.853 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.850+0000 7fdfe62ff700 1 --2- 172.21.15.27:0/2960046768 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdfe0159c50 0x7fdfe015e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:44.853 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.850+0000 7fdfe62ff700 1 -- 172.21.15.27:0/2960046768 >> 172.21.15.27:0/2960046768 conn(0x7fdfe006e910 msgr2=0x7fdfe006be30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:44.854 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.850+0000 7fdfe62ff700 1 -- 172.21.15.27:0/2960046768 shutdown_connections 2023-10-06T01:41:44.854 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.850+0000 7fdfe62ff700 1 -- 172.21.15.27:0/2960046768 wait complete. 2023-10-06T01:41:44.857 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.851+0000 7fdfe62ff700 1 Processor -- start 2023-10-06T01:41:44.857 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.851+0000 7fdfe62ff700 1 -- start start 2023-10-06T01:41:44.857 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.852+0000 7fdfe62ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfe0069120 0x7fdfe00b16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:44.857 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.852+0000 7fdfe62ff700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdfe00694f0 0x7fdfe00c18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:44.857 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.852+0000 7fdfe62ff700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdfe0159c50 0x7fdfe015e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:44.857 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.852+0000 7fdfe62ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe0156af0 con 0x7fdfe0069120 2023-10-06T01:41:44.857 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.852+0000 7fdfe62ff700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe015cfe0 con 0x7fdfe0159c50 2023-10-06T01:41:44.857 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.852+0000 7fdfe62ff700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe015cd00 con 0x7fdfe00694f0 2023-10-06T01:41:44.857 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.852+0000 7fdfdffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfe0069120 0x7fdfe00b16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:44.857 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.852+0000 7fdfdffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfe0069120 0x7fdfe00b16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34486/0 (socket says 172.21.15.27:34486) 2023-10-06T01:41:44.858 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.852+0000 7fdfdffff700 1 -- 172.21.15.27:0/1149651562 learned_addr learned my addr 172.21.15.27:0/1149651562 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:44.858 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.852+0000 7fdfdf7fe700 1 --2- 172.21.15.27:0/1149651562 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdfe00694f0 0x7fdfe00c18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:44.858 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.852+0000 7fdfe489c700 1 --2- 172.21.15.27:0/1149651562 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdfe0159c50 0x7fdfe015e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:44.858 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.852+0000 7fdfdffff700 1 -- 172.21.15.27:0/1149651562 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdfe00694f0 msgr2=0x7fdfe00c18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:44.858 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.852+0000 7fdfdffff700 1 --2- 172.21.15.27:0/1149651562 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdfe00694f0 0x7fdfe00c18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:44.858 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.852+0000 7fdfdffff700 1 -- 172.21.15.27:0/1149651562 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdfe0159c50 msgr2=0x7fdfe015e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:44.858 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.852+0000 7fdfdffff700 1 --2- 172.21.15.27:0/1149651562 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdfe0159c50 0x7fdfe015e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:44.858 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.852+0000 7fdfdffff700 1 -- 172.21.15.27:0/1149651562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd8009a20 con 0x7fdfe0069120 2023-10-06T01:41:44.858 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.852+0000 7fdfdf7fe700 1 --2- 172.21.15.27:0/1149651562 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdfe00694f0 0x7fdfe00c18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:41:44.858 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.852+0000 7fdfe489c700 1 --2- 172.21.15.27:0/1149651562 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdfe0159c50 0x7fdfe015e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:41:44.858 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.853+0000 7fdfdffff700 1 --2- 172.21.15.27:0/1149651562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfe0069120 0x7fdfe00b16f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fdfd000c7c0 tx=0x7fdfd0007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:44.858 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.853+0000 7fdfd7fff700 1 -- 172.21.15.27:0/1149651562 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd0005b30 con 0x7fdfe0069120 2023-10-06T01:41:44.859 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.853+0000 7fdfd7fff700 1 -- 172.21.15.27:0/1149651562 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfd0005c90 con 0x7fdfe0069120 2023-10-06T01:41:44.859 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.853+0000 7fdfe62ff700 1 -- 172.21.15.27:0/1149651562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfe015dc50 con 0x7fdfe0069120 2023-10-06T01:41:44.859 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.853+0000 7fdfd7fff700 1 -- 172.21.15.27:0/1149651562 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd0007340 con 0x7fdfe0069120 2023-10-06T01:41:44.859 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.853+0000 7fdfe62ff700 1 -- 172.21.15.27:0/1149651562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfe015e210 con 0x7fdfe0069120 2023-10-06T01:41:44.859 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.854+0000 7fdfe62ff700 1 -- 172.21.15.27:0/1149651562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfe0064050 con 0x7fdfe0069120 2023-10-06T01:41:44.859 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.854+0000 7fdfd7fff700 1 -- 172.21.15.27:0/1149651562 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fdfd001b400 con 0x7fdfe0069120 2023-10-06T01:41:44.859 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.855+0000 7fdfd7fff700 1 --2- 172.21.15.27:0/1149651562 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdfc0066f30 0x7fdfc00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:44.859 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.855+0000 7fdfd7fff700 1 -- 172.21.15.27:0/1149651562 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fdfd0084130 con 0x7fdfe0069120 2023-10-06T01:41:44.859 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.855+0000 7fdfdf7fe700 1 --2- 172.21.15.27:0/1149651562 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdfc0066f30 0x7fdfc00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:44.859 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.855+0000 7fdfdf7fe700 1 --2- 172.21.15.27:0/1149651562 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdfc0066f30 0x7fdfc00693e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fdfc8006f00 tx=0x7fdfc8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:44.860 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:44.859+0000 7fdfd7fff700 1 -- 172.21.15.27:0/1149651562 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdfd00512c0 con 0x7fdfe0069120 2023-10-06T01:41:45.055 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.054+0000 7fdfe62ff700 1 -- 172.21.15.27:0/1149651562 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fdfe00ae390 con 0x7fdfe0069120 2023-10-06T01:41:45.055 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.054+0000 7fdfd7fff700 1 -- 172.21.15.27:0/1149651562 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7fdfd00548e0 con 0x7fdfe0069120 2023-10-06T01:41:45.055 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 8 2023-10-06T01:41:45.058 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.056+0000 7fdfe62ff700 1 -- 172.21.15.27:0/1149651562 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdfc0066f30 msgr2=0x7fdfc00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:45.058 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.056+0000 7fdfe62ff700 1 --2- 172.21.15.27:0/1149651562 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdfc0066f30 0x7fdfc00693e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fdfc8006f00 tx=0x7fdfc8008040 comp rx=0 tx=0).stop 2023-10-06T01:41:45.058 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.056+0000 7fdfe62ff700 1 -- 172.21.15.27:0/1149651562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfe0069120 msgr2=0x7fdfe00b16f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:45.058 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.056+0000 7fdfe62ff700 1 --2- 172.21.15.27:0/1149651562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfe0069120 0x7fdfe00b16f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fdfd000c7c0 tx=0x7fdfd0007e00 comp rx=0 tx=0).stop 2023-10-06T01:41:45.058 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.056+0000 7fdfe62ff700 1 -- 172.21.15.27:0/1149651562 shutdown_connections 2023-10-06T01:41:45.058 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.056+0000 7fdfe62ff700 1 --2- 172.21.15.27:0/1149651562 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdfc0066f30 0x7fdfc00693e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:45.058 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.056+0000 7fdfe62ff700 1 --2- 172.21.15.27:0/1149651562 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdfe0159c50 0x7fdfe015e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:45.058 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.056+0000 7fdfe62ff700 1 --2- 172.21.15.27:0/1149651562 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdfe0069120 0x7fdfe00b16f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:45.058 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.056+0000 7fdfe62ff700 1 --2- 172.21.15.27:0/1149651562 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdfe00694f0 0x7fdfe00c18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:45.058 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.056+0000 7fdfe62ff700 1 -- 172.21.15.27:0/1149651562 >> 172.21.15.27:0/1149651562 conn(0x7fdfe006e910 msgr2=0x7fdfe0158510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:45.059 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.056+0000 7fdfe62ff700 1 -- 172.21.15.27:0/1149651562 shutdown_connections 2023-10-06T01:41:45.059 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.056+0000 7fdfe62ff700 1 -- 172.21.15.27:0/1149651562 wait complete. 2023-10-06T01:41:45.070 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-06T01:41:45.191 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.190+0000 7f28f6df9700 1 Processor -- start 2023-10-06T01:41:45.191 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.190+0000 7f28f6df9700 1 -- start start 2023-10-06T01:41:45.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.190+0000 7f28f6df9700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f28f0159500 0x7f28f01598d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:45.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.190+0000 7f28f6df9700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f28f00b3fd0 0x7f28f015d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:45.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.190+0000 7f28f6df9700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f28f015d8e0 0x7f28f00cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:45.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.190+0000 7f28f6df9700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28f015dd20 con 0x7f28f0159500 2023-10-06T01:41:45.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.190+0000 7f28f6df9700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28f00b1110 con 0x7f28f00b3fd0 2023-10-06T01:41:45.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.190+0000 7f28f6df9700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28f00b13b0 con 0x7f28f015d8e0 2023-10-06T01:41:45.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.191+0000 7f28f4b95700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f28f0159500 0x7f28f01598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:45.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.191+0000 7f28f4b95700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f28f0159500 0x7f28f01598d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34492/0 (socket says 172.21.15.27:34492) 2023-10-06T01:41:45.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.191+0000 7f28f4b95700 1 -- 172.21.15.27:0/256776104 learned_addr learned my addr 172.21.15.27:0/256776104 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:45.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.191+0000 7f28effff700 1 --2- 172.21.15.27:0/256776104 >> v2:172.21.15.124:3300/0 conn(0x7f28f00b3fd0 0x7f28f015d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:45.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.191+0000 7f28f5396700 1 --2- 172.21.15.27:0/256776104 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f28f015d8e0 0x7f28f00cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:45.195 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.191+0000 7f28f4b95700 1 -- 172.21.15.27:0/256776104 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f28f015d8e0 msgr2=0x7f28f00cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:45.195 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.191+0000 7f28f4b95700 1 --2- 172.21.15.27:0/256776104 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f28f015d8e0 0x7f28f00cf6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:45.195 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.191+0000 7f28f4b95700 1 -- 172.21.15.27:0/256776104 >> v2:172.21.15.124:3300/0 conn(0x7f28f00b3fd0 msgr2=0x7f28f015d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:45.195 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.191+0000 7f28f4b95700 1 --2- 172.21.15.27:0/256776104 >> v2:172.21.15.124:3300/0 conn(0x7f28f00b3fd0 0x7f28f015d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:45.195 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.191+0000 7f28f4b95700 1 -- 172.21.15.27:0/256776104 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28f00b1690 con 0x7f28f0159500 2023-10-06T01:41:45.195 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.191+0000 7f28f4b95700 1 --2- 172.21.15.27:0/256776104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28f0159500 0x7f28f01598d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f28f0063430 tx=0x7f28e8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8006c52578d93a8f server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:45.195 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.191+0000 7f28ef7fe700 1 -- 172.21.15.27:0/256776104 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e800e070 con 0x7f28f0159500 2023-10-06T01:41:45.195 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.191+0000 7f28ef7fe700 1 -- 172.21.15.27:0/256776104 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e800ba50 con 0x7f28f0159500 2023-10-06T01:41:45.195 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.191+0000 7f28ef7fe700 1 -- 172.21.15.27:0/256776104 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e80092e0 con 0x7f28f0159500 2023-10-06T01:41:45.196 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.192+0000 7f28f6df9700 1 -- 172.21.15.27:0/256776104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28f0159500 msgr2=0x7f28f01598d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:45.196 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.192+0000 7f28f6df9700 1 --2- 172.21.15.27:0/256776104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28f0159500 0x7f28f01598d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f28f0063430 tx=0x7f28e8009840 comp rx=0 tx=0).stop 2023-10-06T01:41:45.196 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.192+0000 7f28f6df9700 1 -- 172.21.15.27:0/256776104 shutdown_connections 2023-10-06T01:41:45.196 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.192+0000 7f28f6df9700 1 --2- 172.21.15.27:0/256776104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28f0159500 0x7f28f01598d0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:45.196 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.192+0000 7f28f6df9700 1 --2- 172.21.15.27:0/256776104 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f28f015d8e0 0x7f28f00cf6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:45.196 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.192+0000 7f28f6df9700 1 --2- 172.21.15.27:0/256776104 >> v2:172.21.15.124:3300/0 conn(0x7f28f00b3fd0 0x7f28f015d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:45.196 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.192+0000 7f28f6df9700 1 -- 172.21.15.27:0/256776104 >> 172.21.15.27:0/256776104 conn(0x7f28f0067630 msgr2=0x7f28f0069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:45.196 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.192+0000 7f28f6df9700 1 -- 172.21.15.27:0/256776104 shutdown_connections 2023-10-06T01:41:45.196 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.192+0000 7f28f6df9700 1 -- 172.21.15.27:0/256776104 wait complete. 2023-10-06T01:41:45.199 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.193+0000 7f28f6df9700 1 Processor -- start 2023-10-06T01:41:45.199 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.193+0000 7f28f6df9700 1 -- start start 2023-10-06T01:41:45.199 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.194+0000 7f28f6df9700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f28f00b3fd0 0x7f28f015be50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:45.199 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.194+0000 7f28f6df9700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28f0159500 0x7f28f0151380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:45.199 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.194+0000 7f28f6df9700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f28f015d8e0 0x7f28f0154ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:45.199 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.194+0000 7f28f6df9700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f01507f0 con 0x7f28f0159500 2023-10-06T01:41:45.199 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.194+0000 7f28f6df9700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28f015b9c0 con 0x7f28f015d8e0 2023-10-06T01:41:45.200 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.194+0000 7f28f6df9700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28f015c390 con 0x7f28f00b3fd0 2023-10-06T01:41:45.200 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.194+0000 7f28effff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28f0159500 0x7f28f0151380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:45.200 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.194+0000 7f28effff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28f0159500 0x7f28f0151380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:34502/0 (socket says 172.21.15.27:34502) 2023-10-06T01:41:45.200 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.194+0000 7f28effff700 1 -- 172.21.15.27:0/36246839 learned_addr learned my addr 172.21.15.27:0/36246839 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:45.200 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.194+0000 7f28f5396700 1 --2- 172.21.15.27:0/36246839 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f28f015d8e0 0x7f28f0154ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:45.200 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.194+0000 7f28f4b95700 1 --2- 172.21.15.27:0/36246839 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f28f00b3fd0 0x7f28f015be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:45.200 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.194+0000 7f28effff700 1 -- 172.21.15.27:0/36246839 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f28f00b3fd0 msgr2=0x7f28f015be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:45.200 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.194+0000 7f28effff700 1 --2- 172.21.15.27:0/36246839 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f28f00b3fd0 0x7f28f015be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:45.200 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.194+0000 7f28effff700 1 -- 172.21.15.27:0/36246839 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f28f015d8e0 msgr2=0x7f28f0154ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:45.200 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.194+0000 7f28effff700 1 --2- 172.21.15.27:0/36246839 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f28f015d8e0 0x7f28f0154ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:45.200 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.194+0000 7f28effff700 1 -- 172.21.15.27:0/36246839 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e8009ae0 con 0x7f28f0159500 2023-10-06T01:41:45.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.195+0000 7f28effff700 1 --2- 172.21.15.27:0/36246839 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28f0159500 0x7f28f0151380 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f28e000c7d0 tx=0x7f28e0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:45.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.195+0000 7f28ecff9700 1 -- 172.21.15.27:0/36246839 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e0011600 con 0x7f28f0159500 2023-10-06T01:41:45.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.195+0000 7f28ecff9700 1 -- 172.21.15.27:0/36246839 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28e0011760 con 0x7f28f0159500 2023-10-06T01:41:45.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.195+0000 7f28f6df9700 1 -- 172.21.15.27:0/36246839 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28f015add0 con 0x7f28f0159500 2023-10-06T01:41:45.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.195+0000 7f28ecff9700 1 -- 172.21.15.27:0/36246839 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e0011b20 con 0x7f28f0159500 2023-10-06T01:41:45.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.195+0000 7f28f6df9700 1 -- 172.21.15.27:0/36246839 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28f015b3f0 con 0x7f28f0159500 2023-10-06T01:41:45.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.196+0000 7f28f6df9700 1 -- 172.21.15.27:0/36246839 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28f015bb50 con 0x7f28f0159500 2023-10-06T01:41:45.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.196+0000 7f28ecff9700 1 -- 172.21.15.27:0/36246839 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f28e00193f0 con 0x7f28f0159500 2023-10-06T01:41:45.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.196+0000 7f28ecff9700 1 --2- 172.21.15.27:0/36246839 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f28d0066f30 0x7f28d00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:45.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.197+0000 7f28ecff9700 1 -- 172.21.15.27:0/36246839 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f28e0084aa0 con 0x7f28f0159500 2023-10-06T01:41:45.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.197+0000 7f28f4b95700 1 --2- 172.21.15.27:0/36246839 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f28d0066f30 0x7f28d00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:45.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.197+0000 7f28f4b95700 1 --2- 172.21.15.27:0/36246839 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f28d0066f30 0x7f28d00693e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f28e8009880 tx=0x7f28e8018000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:45.202 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.201+0000 7f28ecff9700 1 -- 172.21.15.27:0/36246839 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f28e0051bb0 con 0x7f28f0159500 2023-10-06T01:41:45.398 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.396+0000 7f28f6df9700 1 -- 172.21.15.27:0/36246839 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f28f0064050 con 0x7f28f0159500 2023-10-06T01:41:45.398 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.396+0000 7f28ecff9700 1 -- 172.21.15.27:0/36246839 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f28e00551d0 con 0x7f28f0159500 2023-10-06T01:41:45.398 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2023-10-06T01:41:45.400 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.398+0000 7f28f6df9700 1 -- 172.21.15.27:0/36246839 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f28d0066f30 msgr2=0x7f28d00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:45.400 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.398+0000 7f28f6df9700 1 --2- 172.21.15.27:0/36246839 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f28d0066f30 0x7f28d00693e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f28e8009880 tx=0x7f28e8018000 comp rx=0 tx=0).stop 2023-10-06T01:41:45.400 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.398+0000 7f28f6df9700 1 -- 172.21.15.27:0/36246839 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28f0159500 msgr2=0x7f28f0151380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:45.400 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.398+0000 7f28f6df9700 1 --2- 172.21.15.27:0/36246839 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28f0159500 0x7f28f0151380 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f28e000c7d0 tx=0x7f28e0008040 comp rx=0 tx=0).stop 2023-10-06T01:41:45.400 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.398+0000 7f28f6df9700 1 -- 172.21.15.27:0/36246839 shutdown_connections 2023-10-06T01:41:45.400 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.398+0000 7f28f6df9700 1 --2- 172.21.15.27:0/36246839 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f28d0066f30 0x7f28d00693e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:45.401 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.398+0000 7f28f6df9700 1 --2- 172.21.15.27:0/36246839 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f28f00b3fd0 0x7f28f015be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:45.401 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.398+0000 7f28f6df9700 1 --2- 172.21.15.27:0/36246839 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f28f015d8e0 0x7f28f0154ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:45.401 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.398+0000 7f28f6df9700 1 --2- 172.21.15.27:0/36246839 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f28f0159500 0x7f28f0151380 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:45.401 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.398+0000 7f28f6df9700 1 -- 172.21.15.27:0/36246839 >> 172.21.15.27:0/36246839 conn(0x7f28f0067630 msgr2=0x7f28f0069a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:45.401 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.399+0000 7f28f6df9700 1 -- 172.21.15.27:0/36246839 shutdown_connections 2023-10-06T01:41:45.401 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.399+0000 7f28f6df9700 1 -- 172.21.15.27:0/36246839 wait complete. 2023-10-06T01:41:45.412 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-06T01:41:45.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.533+0000 7f444359e700 1 Processor -- start 2023-10-06T01:41:45.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.533+0000 7f444359e700 1 -- start start 2023-10-06T01:41:45.537 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.534+0000 7f444359e700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f4444142e90 0x7f4444069810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:45.537 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.534+0000 7f444359e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4444069d50 0x7f444415d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:45.537 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.534+0000 7f444359e700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4444069380 con 0x7f444415de80 2023-10-06T01:41:45.537 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.534+0000 7f444359e700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4444068770 con 0x7f4444142e90 2023-10-06T01:41:45.537 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.534+0000 7f444359e700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4444068a50 con 0x7f4444069d50 2023-10-06T01:41:45.538 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.534+0000 7f4442d9d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f444415de80 0x7f44440b0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38432/0 (socket says 172.21.15.27:38432) 2023-10-06T01:41:45.538 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.534+0000 7f4442d9d700 1 -- 172.21.15.27:0/3724185195 learned_addr learned my addr 172.21.15.27:0/3724185195 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:45.538 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.534+0000 7f4441d9b700 1 --2- 172.21.15.27:0/3724185195 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4444069d50 0x7f444415d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:45.538 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.534+0000 7f444259c700 1 --2- 172.21.15.27:0/3724185195 >> v2:172.21.15.124:3300/0 conn(0x7f4444142e90 0x7f4444069810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:45.538 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.534+0000 7f444159a700 1 -- 172.21.15.27:0/3724185195 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 357646111 0 0) 0x7f4444069380 con 0x7f444415de80 2023-10-06T01:41:45.538 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.534+0000 7f444159a700 1 -- 172.21.15.27:0/3724185195 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f442c003570 con 0x7f444415de80 2023-10-06T01:41:45.538 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.534+0000 7f444159a700 1 -- 172.21.15.27:0/3724185195 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 164718050 0 0) 0x7f442c003570 con 0x7f444415de80 2023-10-06T01:41:45.538 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.534+0000 7f4441d9b700 1 -- 172.21.15.27:0/3724185195 >> v2:172.21.15.124:3300/0 conn(0x7f4444142e90 msgr2=0x7f4444069810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:45.538 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.534+0000 7f4441d9b700 1 --2- 172.21.15.27:0/3724185195 >> v2:172.21.15.124:3300/0 conn(0x7f4444142e90 0x7f4444069810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:45.538 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.534+0000 7f4441d9b700 1 -- 172.21.15.27:0/3724185195 >> v1:172.21.15.27:6789/0 conn(0x7f444415de80 legacy=0x7f44440b0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:45.538 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.534+0000 7f4441d9b700 1 -- 172.21.15.27:0/3724185195 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4444068d30 con 0x7f4444069d50 2023-10-06T01:41:45.539 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.534+0000 7f444259c700 1 --2- 172.21.15.27:0/3724185195 >> v2:172.21.15.124:3300/0 conn(0x7f4444142e90 0x7f4444069810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:41:45.539 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.535+0000 7f4441d9b700 1 --2- 172.21.15.27:0/3724185195 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4444069d50 0x7f444415d290 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f44380098e0 tx=0x7f44380091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a9c6fc25e6bd3a server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:45.539 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.535+0000 7f444159a700 1 -- 172.21.15.27:0/3724185195 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443c0017f0 con 0x7f4444069d50 2023-10-06T01:41:45.539 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.535+0000 7f444159a700 1 -- 172.21.15.27:0/3724185195 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4438011420 con 0x7f4444069d50 2023-10-06T01:41:45.539 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.535+0000 7f444159a700 1 -- 172.21.15.27:0/3724185195 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44380117c0 con 0x7f4444069d50 2023-10-06T01:41:45.539 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.535+0000 7f444359e700 1 -- 172.21.15.27:0/3724185195 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4444069d50 msgr2=0x7f444415d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:45.539 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.535+0000 7f444359e700 1 --2- 172.21.15.27:0/3724185195 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4444069d50 0x7f444415d290 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f44380098e0 tx=0x7f44380091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:45.539 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.535+0000 7f444359e700 1 -- 172.21.15.27:0/3724185195 shutdown_connections 2023-10-06T01:41:45.539 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.535+0000 7f444359e700 1 --2- 172.21.15.27:0/3724185195 >> v2:172.21.15.124:3300/0 conn(0x7f4444142e90 0x7f4444069810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:45.539 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.535+0000 7f444359e700 1 --2- 172.21.15.27:0/3724185195 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4444069d50 0x7f444415d290 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:45.539 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.535+0000 7f444359e700 1 -- 172.21.15.27:0/3724185195 >> 172.21.15.27:0/3724185195 conn(0x7f44440ccfe0 msgr2=0x7f44440cd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:45.540 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.535+0000 7f444359e700 1 -- 172.21.15.27:0/3724185195 shutdown_connections 2023-10-06T01:41:45.540 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.535+0000 7f444359e700 1 -- 172.21.15.27:0/3724185195 wait complete. 2023-10-06T01:41:45.542 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.537+0000 7f444359e700 1 Processor -- start 2023-10-06T01:41:45.542 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.537+0000 7f444359e700 1 -- start start 2023-10-06T01:41:45.543 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.537+0000 7f444359e700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4444069d50 0x7f44440c8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:45.543 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.537+0000 7f444359e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4444142e90 0x7f444416d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:45.543 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.537+0000 7f444359e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f444415de80 0x7f444413ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:45.543 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.537+0000 7f444359e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44440c62e0 con 0x7f444415de80 2023-10-06T01:41:45.543 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.537+0000 7f444359e700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44440c8bd0 con 0x7f4444069d50 2023-10-06T01:41:45.543 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.537+0000 7f444359e700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44440c65a0 con 0x7f4444142e90 2023-10-06T01:41:45.543 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.537+0000 7f4442d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f444415de80 0x7f444413ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:45.543 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.537+0000 7f4442d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f444415de80 0x7f444413ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39388/0 (socket says 172.21.15.27:39388) 2023-10-06T01:41:45.544 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.538+0000 7f4442d9d700 1 -- 172.21.15.27:0/1917080983 learned_addr learned my addr 172.21.15.27:0/1917080983 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:45.544 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.538+0000 7f4441d9b700 1 --2- 172.21.15.27:0/1917080983 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4444142e90 0x7f444416d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:45.544 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.538+0000 7f444259c700 1 --2- 172.21.15.27:0/1917080983 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4444069d50 0x7f44440c8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:45.544 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.538+0000 7f4442d9d700 1 -- 172.21.15.27:0/1917080983 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4444142e90 msgr2=0x7f444416d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:45.544 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.538+0000 7f4442d9d700 1 --2- 172.21.15.27:0/1917080983 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4444142e90 0x7f444416d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:45.545 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.538+0000 7f4442d9d700 1 -- 172.21.15.27:0/1917080983 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4444069d50 msgr2=0x7f44440c8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:45.545 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.538+0000 7f4442d9d700 1 --2- 172.21.15.27:0/1917080983 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4444069d50 0x7f44440c8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:45.545 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.538+0000 7f4442d9d700 1 -- 172.21.15.27:0/1917080983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4434009420 con 0x7f444415de80 2023-10-06T01:41:45.545 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.538+0000 7f4442d9d700 1 --2- 172.21.15.27:0/1917080983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f444415de80 0x7f444413ee50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f443c00cb90 tx=0x7f443c001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:45.545 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.538+0000 7f44327fc700 1 -- 172.21.15.27:0/1917080983 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443c004dc0 con 0x7f444415de80 2023-10-06T01:41:45.545 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.538+0000 7f44327fc700 1 -- 172.21.15.27:0/1917080983 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f443c00a5d0 con 0x7f444415de80 2023-10-06T01:41:45.545 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.538+0000 7f444359e700 1 -- 172.21.15.27:0/1917080983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4438009420 con 0x7f444415de80 2023-10-06T01:41:45.545 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.538+0000 7f44327fc700 1 -- 172.21.15.27:0/1917080983 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443c00a990 con 0x7f444415de80 2023-10-06T01:41:45.545 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.538+0000 7f444359e700 1 -- 172.21.15.27:0/1917080983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44440c6b60 con 0x7f444415de80 2023-10-06T01:41:45.545 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.539+0000 7f444359e700 1 -- 172.21.15.27:0/1917080983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f444404c3c0 con 0x7f444415de80 2023-10-06T01:41:45.545 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.540+0000 7f44327fc700 1 -- 172.21.15.27:0/1917080983 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f443c007150 con 0x7f444415de80 2023-10-06T01:41:45.545 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.540+0000 7f44327fc700 1 --2- 172.21.15.27:0/1917080983 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4424066f30 0x7f44240693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:45.546 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.540+0000 7f44327fc700 1 -- 172.21.15.27:0/1917080983 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f443c084060 con 0x7f444415de80 2023-10-06T01:41:45.546 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.540+0000 7f444259c700 1 --2- 172.21.15.27:0/1917080983 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4424066f30 0x7f44240693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:45.546 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.541+0000 7f444259c700 1 --2- 172.21.15.27:0/1917080983 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4424066f30 0x7f44240693e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4434009f40 tx=0x7f44340091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:45.546 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.544+0000 7f44327fc700 1 -- 172.21.15.27:0/1917080983 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f443c0511f0 con 0x7f444415de80 2023-10-06T01:41:45.736 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.735+0000 7f444359e700 1 -- 172.21.15.27:0/1917080983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4444069010 con 0x7f444415de80 2023-10-06T01:41:45.737 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.735+0000 7f44327fc700 1 -- 172.21.15.27:0/1917080983 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f443c054810 con 0x7f444415de80 2023-10-06T01:41:45.737 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2023-10-06T01:41:45.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.737+0000 7f444359e700 1 -- 172.21.15.27:0/1917080983 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4424066f30 msgr2=0x7f44240693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:45.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.737+0000 7f444359e700 1 --2- 172.21.15.27:0/1917080983 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4424066f30 0x7f44240693e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4434009f40 tx=0x7f44340091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:45.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.737+0000 7f444359e700 1 -- 172.21.15.27:0/1917080983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f444415de80 msgr2=0x7f444413ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:45.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.737+0000 7f444359e700 1 --2- 172.21.15.27:0/1917080983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f444415de80 0x7f444413ee50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f443c00cb90 tx=0x7f443c001640 comp rx=0 tx=0).stop 2023-10-06T01:41:45.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.737+0000 7f444359e700 1 -- 172.21.15.27:0/1917080983 shutdown_connections 2023-10-06T01:41:45.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.737+0000 7f444359e700 1 --2- 172.21.15.27:0/1917080983 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4424066f30 0x7f44240693e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:45.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.737+0000 7f444359e700 1 --2- 172.21.15.27:0/1917080983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f444415de80 0x7f444413ee50 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:45.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.737+0000 7f444359e700 1 --2- 172.21.15.27:0/1917080983 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4444069d50 0x7f44440c8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:45.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.737+0000 7f444359e700 1 --2- 172.21.15.27:0/1917080983 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4444142e90 0x7f444416d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:45.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.737+0000 7f444359e700 1 -- 172.21.15.27:0/1917080983 >> 172.21.15.27:0/1917080983 conn(0x7f44440ccfe0 msgr2=0x7f444415c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:45.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.737+0000 7f444359e700 1 -- 172.21.15.27:0/1917080983 shutdown_connections 2023-10-06T01:41:45.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.738+0000 7f444359e700 1 -- 172.21.15.27:0/1917080983 wait complete. 2023-10-06T01:41:45.751 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-06T01:41:45.871 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.870+0000 7fe2bcf0d700 1 Processor -- start 2023-10-06T01:41:45.872 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.870+0000 7fe2bcf0d700 1 -- start start 2023-10-06T01:41:45.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.871+0000 7fe2bcf0d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe2b8159500 0x7fe2b81598d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:45.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.871+0000 7fe2bcf0d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe2b80b3fd0 0x7fe2b815d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:45.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.871+0000 7fe2bcf0d700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe2b815d8e0 0x7fe2b80cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:45.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.871+0000 7fe2bcf0d700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2b815dd20 con 0x7fe2b80b3fd0 2023-10-06T01:41:45.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.871+0000 7fe2bcf0d700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2b80b1110 con 0x7fe2b815d8e0 2023-10-06T01:41:45.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.871+0000 7fe2bcf0d700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b80b13b0 con 0x7fe2b8159500 2023-10-06T01:41:45.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.871+0000 7fe2b5d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe2b80b3fd0 0x7fe2b815d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:45.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.871+0000 7fe2b5d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe2b80b3fd0 0x7fe2b815d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39404/0 (socket says 172.21.15.27:39404) 2023-10-06T01:41:45.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.871+0000 7fe2b5d9b700 1 -- 172.21.15.27:0/2198261925 learned_addr learned my addr 172.21.15.27:0/2198261925 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:45.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.871+0000 7fe2b659c700 1 --2- 172.21.15.27:0/2198261925 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe2b8159500 0x7fe2b81598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:45.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.872+0000 7fe2b6d9d700 1 --2- 172.21.15.27:0/2198261925 >> v2:172.21.15.124:3300/0 conn(0x7fe2b815d8e0 0x7fe2b80cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:45.876 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.872+0000 7fe2b5d9b700 1 -- 172.21.15.27:0/2198261925 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe2b8159500 msgr2=0x7fe2b81598d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:45.876 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.872+0000 7fe2b5d9b700 1 --2- 172.21.15.27:0/2198261925 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe2b8159500 0x7fe2b81598d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:45.876 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.872+0000 7fe2b5d9b700 1 -- 172.21.15.27:0/2198261925 >> v2:172.21.15.124:3300/0 conn(0x7fe2b815d8e0 msgr2=0x7fe2b80cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:45.876 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.872+0000 7fe2b5d9b700 1 --2- 172.21.15.27:0/2198261925 >> v2:172.21.15.124:3300/0 conn(0x7fe2b815d8e0 0x7fe2b80cf6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:45.876 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.872+0000 7fe2b5d9b700 1 -- 172.21.15.27:0/2198261925 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2b80b1690 con 0x7fe2b80b3fd0 2023-10-06T01:41:45.876 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.872+0000 7fe2b6d9d700 1 --2- 172.21.15.27:0/2198261925 >> v2:172.21.15.124:3300/0 conn(0x7fe2b815d8e0 0x7fe2b80cf6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:41:45.876 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.872+0000 7fe2b659c700 1 --2- 172.21.15.27:0/2198261925 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe2b8159500 0x7fe2b81598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:41:45.876 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.872+0000 7fe2b5d9b700 1 --2- 172.21.15.27:0/2198261925 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2b80b3fd0 0x7fe2b815d3a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe2b80b2f00 tx=0x7fe2a0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56c2cda4c38c404a server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:45.876 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.872+0000 7fe2b559a700 1 -- 172.21.15.27:0/2198261925 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a000b860 con 0x7fe2b80b3fd0 2023-10-06T01:41:45.876 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.872+0000 7fe2b559a700 1 -- 172.21.15.27:0/2198261925 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2a000b9c0 con 0x7fe2b80b3fd0 2023-10-06T01:41:45.877 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.872+0000 7fe2b559a700 1 -- 172.21.15.27:0/2198261925 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2a000bd60 con 0x7fe2b80b3fd0 2023-10-06T01:41:45.877 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.872+0000 7fe2bcf0d700 1 -- 172.21.15.27:0/2198261925 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2b80b3fd0 msgr2=0x7fe2b815d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:45.877 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.872+0000 7fe2bcf0d700 1 --2- 172.21.15.27:0/2198261925 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2b80b3fd0 0x7fe2b815d3a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe2b80b2f00 tx=0x7fe2a0009250 comp rx=0 tx=0).stop 2023-10-06T01:41:45.877 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.873+0000 7fe2bcf0d700 1 -- 172.21.15.27:0/2198261925 shutdown_connections 2023-10-06T01:41:45.877 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.873+0000 7fe2bcf0d700 1 --2- 172.21.15.27:0/2198261925 >> v2:172.21.15.124:3300/0 conn(0x7fe2b815d8e0 0x7fe2b80cf6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:45.877 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.873+0000 7fe2bcf0d700 1 --2- 172.21.15.27:0/2198261925 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe2b8159500 0x7fe2b81598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:45.877 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.873+0000 7fe2bcf0d700 1 --2- 172.21.15.27:0/2198261925 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2b80b3fd0 0x7fe2b815d3a0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:45.877 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.873+0000 7fe2bcf0d700 1 -- 172.21.15.27:0/2198261925 >> 172.21.15.27:0/2198261925 conn(0x7fe2b8067630 msgr2=0x7fe2b8069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:45.877 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.873+0000 7fe2bcf0d700 1 -- 172.21.15.27:0/2198261925 shutdown_connections 2023-10-06T01:41:45.877 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.873+0000 7fe2bcf0d700 1 -- 172.21.15.27:0/2198261925 wait complete. 2023-10-06T01:41:45.877 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.874+0000 7fe2bcf0d700 1 Processor -- start 2023-10-06T01:41:45.878 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.874+0000 7fe2bcf0d700 1 -- start start 2023-10-06T01:41:45.880 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.874+0000 7fe2bcf0d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2b80b3fd0 0x7fe2b8150f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:45.880 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.874+0000 7fe2bcf0d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe2b8159500 0x7fe2b813ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:45.880 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.874+0000 7fe2bcf0d700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe2b815d8e0 0x7fe2b8150070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:45.880 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.874+0000 7fe2bcf0d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b814efc0 con 0x7fe2b80b3fd0 2023-10-06T01:41:45.881 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.874+0000 7fe2bcf0d700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b814f2a0 con 0x7fe2b815d8e0 2023-10-06T01:41:45.881 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.874+0000 7fe2bcf0d700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2b814f580 con 0x7fe2b8159500 2023-10-06T01:41:45.881 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.874+0000 7fe2b659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2b80b3fd0 0x7fe2b8150f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:45.881 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.874+0000 7fe2b659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2b80b3fd0 0x7fe2b8150f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39408/0 (socket says 172.21.15.27:39408) 2023-10-06T01:41:45.881 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.874+0000 7fe2b659c700 1 -- 172.21.15.27:0/3907264281 learned_addr learned my addr 172.21.15.27:0/3907264281 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:45.881 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.875+0000 7fe2b5d9b700 1 --2- 172.21.15.27:0/3907264281 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe2b8159500 0x7fe2b813ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:45.881 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.875+0000 7fe2b6d9d700 1 --2- 172.21.15.27:0/3907264281 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe2b815d8e0 0x7fe2b8150070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:45.881 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.875+0000 7fe2b659c700 1 -- 172.21.15.27:0/3907264281 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe2b8159500 msgr2=0x7fe2b813ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:45.881 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.875+0000 7fe2b659c700 1 --2- 172.21.15.27:0/3907264281 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe2b8159500 0x7fe2b813ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:45.881 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.875+0000 7fe2b659c700 1 -- 172.21.15.27:0/3907264281 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe2b815d8e0 msgr2=0x7fe2b8150070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:45.881 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.875+0000 7fe2b659c700 1 --2- 172.21.15.27:0/3907264281 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe2b815d8e0 0x7fe2b8150070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:45.881 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.875+0000 7fe2b659c700 1 -- 172.21.15.27:0/3907264281 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2a00094f0 con 0x7fe2b80b3fd0 2023-10-06T01:41:45.882 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.875+0000 7fe2b659c700 1 --2- 172.21.15.27:0/3907264281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2b80b3fd0 0x7fe2b8150f00 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe2b000c880 tx=0x7fe2b0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:45.882 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.875+0000 7fe2ae7fc700 1 -- 172.21.15.27:0/3907264281 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b0011600 con 0x7fe2b80b3fd0 2023-10-06T01:41:45.882 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.875+0000 7fe2ae7fc700 1 -- 172.21.15.27:0/3907264281 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b0011760 con 0x7fe2b80b3fd0 2023-10-06T01:41:45.882 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.875+0000 7fe2bcf0d700 1 -- 172.21.15.27:0/3907264281 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2b814f860 con 0x7fe2b80b3fd0 2023-10-06T01:41:45.882 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.875+0000 7fe2ae7fc700 1 -- 172.21.15.27:0/3907264281 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b0011ae0 con 0x7fe2b80b3fd0 2023-10-06T01:41:45.882 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.875+0000 7fe2bcf0d700 1 -- 172.21.15.27:0/3907264281 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2b813f4c0 con 0x7fe2b80b3fd0 2023-10-06T01:41:45.882 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.876+0000 7fe2bcf0d700 1 -- 172.21.15.27:0/3907264281 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2b8064050 con 0x7fe2b80b3fd0 2023-10-06T01:41:45.882 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.877+0000 7fe2ae7fc700 1 -- 172.21.15.27:0/3907264281 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe2b0010350 con 0x7fe2b80b3fd0 2023-10-06T01:41:45.882 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.877+0000 7fe2ae7fc700 1 --2- 172.21.15.27:0/3907264281 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe298066f30 0x7fe2980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:45.882 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.877+0000 7fe2ae7fc700 1 -- 172.21.15.27:0/3907264281 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fe2b0084630 con 0x7fe2b80b3fd0 2023-10-06T01:41:45.883 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.877+0000 7fe2b5d9b700 1 --2- 172.21.15.27:0/3907264281 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe298066f30 0x7fe2980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:45.883 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.878+0000 7fe2b5d9b700 1 --2- 172.21.15.27:0/3907264281 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe298066f30 0x7fe2980693e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe2b80b2f00 tx=0x7fe2a0017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:45.883 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:45.881+0000 7fe2ae7fc700 1 -- 172.21.15.27:0/3907264281 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe2b00551d0 con 0x7fe2b80b3fd0 2023-10-06T01:41:46.074 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.072+0000 7fe2bcf0d700 1 -- 172.21.15.27:0/3907264281 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe2b80b1970 con 0x7fe2b80b3fd0 2023-10-06T01:41:46.074 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.072+0000 7fe2ae7fc700 1 -- 172.21.15.27:0/3907264281 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe2b0054d60 con 0x7fe2b80b3fd0 2023-10-06T01:41:46.075 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2023-10-06T01:41:46.076 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.074+0000 7fe2bcf0d700 1 -- 172.21.15.27:0/3907264281 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe298066f30 msgr2=0x7fe2980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:46.076 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.074+0000 7fe2bcf0d700 1 --2- 172.21.15.27:0/3907264281 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe298066f30 0x7fe2980693e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fe2b80b2f00 tx=0x7fe2a0017000 comp rx=0 tx=0).stop 2023-10-06T01:41:46.076 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.074+0000 7fe2bcf0d700 1 -- 172.21.15.27:0/3907264281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2b80b3fd0 msgr2=0x7fe2b8150f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:46.077 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.074+0000 7fe2bcf0d700 1 --2- 172.21.15.27:0/3907264281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2b80b3fd0 0x7fe2b8150f00 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe2b000c880 tx=0x7fe2b0008040 comp rx=0 tx=0).stop 2023-10-06T01:41:46.077 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.075+0000 7fe2bcf0d700 1 -- 172.21.15.27:0/3907264281 shutdown_connections 2023-10-06T01:41:46.077 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.075+0000 7fe2bcf0d700 1 --2- 172.21.15.27:0/3907264281 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe298066f30 0x7fe2980693e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:46.077 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.075+0000 7fe2bcf0d700 1 --2- 172.21.15.27:0/3907264281 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe2b815d8e0 0x7fe2b8150070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:46.077 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.075+0000 7fe2bcf0d700 1 --2- 172.21.15.27:0/3907264281 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe2b80b3fd0 0x7fe2b8150f00 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:46.078 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.075+0000 7fe2bcf0d700 1 --2- 172.21.15.27:0/3907264281 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe2b8159500 0x7fe2b813ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:46.078 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.075+0000 7fe2bcf0d700 1 -- 172.21.15.27:0/3907264281 >> 172.21.15.27:0/3907264281 conn(0x7fe2b8067630 msgr2=0x7fe2b814b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:46.078 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.075+0000 7fe2bcf0d700 1 -- 172.21.15.27:0/3907264281 shutdown_connections 2023-10-06T01:41:46.078 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.075+0000 7fe2bcf0d700 1 -- 172.21.15.27:0/3907264281 wait complete. 2023-10-06T01:41:46.117 INFO:tasks.cephfs_test_runner:Starting test: test_backtrace (tasks.cephfs.test_backtrace.TestBacktrace) That the 'parent' 'layout' and 'symlink' xattrs on the head objects of files 2023-10-06T01:41:46.117 INFO:tasks.cephfs_test_runner:test_backtrace (tasks.cephfs.test_backtrace.TestBacktrace) 2023-10-06T01:41:46.117 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_backtrace.TestBacktrace.test_backtrace' 2023-10-06T01:41:46.238 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.236+0000 7fa61d3f3700 1 Processor -- start 2023-10-06T01:41:46.238 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.236+0000 7fa61d3f3700 1 -- start start 2023-10-06T01:41:46.245 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.237+0000 7fa61d3f3700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fa61815a850 0x7fa6180b1830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:46.245 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.237+0000 7fa61d3f3700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6180b2ad0 con 0x7fa618150900 2023-10-06T01:41:46.246 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.237+0000 7fa61d3f3700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6180b1ec0 con 0x7fa618150cd0 2023-10-06T01:41:46.246 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.237+0000 7fa61d3f3700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6180699a0 con 0x7fa61815a850 2023-10-06T01:41:46.246 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.237+0000 7fa616ffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa618150900 0x7fa61815a140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38438/0 (socket says 172.21.15.27:38438) 2023-10-06T01:41:46.246 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.237+0000 7fa616ffd700 1 -- 172.21.15.27:0/620892362 learned_addr learned my addr 172.21.15.27:0/620892362 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:46.247 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.237+0000 7fa615ffb700 1 -- 172.21.15.27:0/620892362 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2341307639 0 0) 0x7fa6180b2ad0 con 0x7fa618150900 2023-10-06T01:41:46.247 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.237+0000 7fa615ffb700 1 -- 172.21.15.27:0/620892362 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa604003570 con 0x7fa618150900 2023-10-06T01:41:46.247 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.238+0000 7fa6167fc700 1 --2- 172.21.15.27:0/620892362 >> v2:172.21.15.124:3300/0 conn(0x7fa61815a850 0x7fa6180b1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:46.247 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.238+0000 7fa615ffb700 1 -- 172.21.15.27:0/620892362 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 325949139 0 0) 0x7fa604003570 con 0x7fa618150900 2023-10-06T01:41:46.247 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.238+0000 7fa615ffb700 1 -- 172.21.15.27:0/620892362 >> v2:172.21.15.124:3300/0 conn(0x7fa61815a850 msgr2=0x7fa6180b1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:46.247 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.238+0000 7fa615ffb700 1 --2- 172.21.15.27:0/620892362 >> v2:172.21.15.124:3300/0 conn(0x7fa61815a850 0x7fa6180b1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:46.247 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.238+0000 7fa615ffb700 1 -- 172.21.15.27:0/620892362 >> v1:172.21.15.124:6789/0 conn(0x7fa618150cd0 legacy=0x7fa61806c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:46.247 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.238+0000 7fa615ffb700 1 -- 172.21.15.27:0/620892362 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6180b21a0 con 0x7fa618150900 2023-10-06T01:41:46.248 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.238+0000 7fa615ffb700 1 -- 172.21.15.27:0/620892362 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2619078944 0 0) 0x7fa610001470 con 0x7fa618150900 2023-10-06T01:41:46.248 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.238+0000 7fa615ffb700 1 -- 172.21.15.27:0/620892362 >> v1:172.21.15.27:6789/0 conn(0x7fa618150900 legacy=0x7fa61815a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:46.248 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.238+0000 7fa615ffb700 1 --2- 172.21.15.27:0/620892362 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa604003ab0 0x7fa604003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:46.248 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.238+0000 7fa615ffb700 1 --2- 172.21.15.27:0/620892362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa604004810 0x7fa604004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:46.248 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.238+0000 7fa615ffb700 1 --2- 172.21.15.27:0/620892362 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa604005210 0x7fa60400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:46.248 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.238+0000 7fa615ffb700 1 -- 172.21.15.27:0/620892362 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa60400c180 con 0x7fa604004810 2023-10-06T01:41:46.248 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.238+0000 7fa615ffb700 1 -- 172.21.15.27:0/620892362 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa60400c2f0 con 0x7fa604003ab0 2023-10-06T01:41:46.249 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.238+0000 7fa615ffb700 1 -- 172.21.15.27:0/620892362 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa60400c4b0 con 0x7fa604005210 2023-10-06T01:41:46.249 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.238+0000 7fa6167fc700 1 --2- 172.21.15.27:0/620892362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa604004810 0x7fa604004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:46.249 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.238+0000 7fa616ffd700 1 --2- 172.21.15.27:0/620892362 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa604003ab0 0x7fa604003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:46.249 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.238+0000 7fa6177fe700 1 --2- 172.21.15.27:0/620892362 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa604005210 0x7fa60400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:46.249 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.238+0000 7fa6167fc700 1 -- 172.21.15.27:0/620892362 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa604005210 msgr2=0x7fa60400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:46.249 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.238+0000 7fa6167fc700 1 --2- 172.21.15.27:0/620892362 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa604005210 0x7fa60400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:46.249 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.238+0000 7fa6167fc700 1 -- 172.21.15.27:0/620892362 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa604003ab0 msgr2=0x7fa604003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:46.250 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.238+0000 7fa6167fc700 1 --2- 172.21.15.27:0/620892362 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa604003ab0 0x7fa604003ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:46.250 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.238+0000 7fa6167fc700 1 -- 172.21.15.27:0/620892362 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa604003570 con 0x7fa604004810 2023-10-06T01:41:46.250 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.238+0000 7fa6177fe700 1 --2- 172.21.15.27:0/620892362 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa604005210 0x7fa60400ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:41:46.250 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.238+0000 7fa6177fe700 1 -- 172.21.15.27:0/620892362 reap_dead start 2023-10-06T01:41:46.250 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.238+0000 7fa6167fc700 1 --2- 172.21.15.27:0/620892362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa604004810 0x7fa604004c80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa61815ac50 tx=0x7fa6000091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86bd1e37b923784b server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:46.250 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.239+0000 7fa615ffb700 1 -- 172.21.15.27:0/620892362 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60000e710 con 0x7fa604004810 2023-10-06T01:41:46.250 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.239+0000 7fa615ffb700 1 -- 172.21.15.27:0/620892362 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa60000e870 con 0x7fa604004810 2023-10-06T01:41:46.251 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.239+0000 7fa615ffb700 1 -- 172.21.15.27:0/620892362 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa60000ec30 con 0x7fa604004810 2023-10-06T01:41:46.251 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.239+0000 7fa61d3f3700 1 -- 172.21.15.27:0/620892362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa604004810 msgr2=0x7fa604004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:46.251 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.239+0000 7fa61d3f3700 1 --2- 172.21.15.27:0/620892362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa604004810 0x7fa604004c80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa61815ac50 tx=0x7fa6000091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:46.251 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.239+0000 7fa61d3f3700 1 -- 172.21.15.27:0/620892362 shutdown_connections 2023-10-06T01:41:46.251 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.239+0000 7fa61d3f3700 1 --2- 172.21.15.27:0/620892362 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa604004810 0x7fa604004c80 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:46.251 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.239+0000 7fa61d3f3700 1 -- 172.21.15.27:0/620892362 >> 172.21.15.27:0/620892362 conn(0x7fa61806e520 msgr2=0x7fa61806e920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:46.252 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.239+0000 7fa61d3f3700 1 -- 172.21.15.27:0/620892362 shutdown_connections 2023-10-06T01:41:46.252 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.239+0000 7fa61d3f3700 1 -- 172.21.15.27:0/620892362 wait complete. 2023-10-06T01:41:46.252 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.240+0000 7fa61d3f3700 1 Processor -- start 2023-10-06T01:41:46.252 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.240+0000 7fa61d3f3700 1 -- start start 2023-10-06T01:41:46.252 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.241+0000 7fa61d3f3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa604004810 0x7fa61814ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:46.252 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.241+0000 7fa61d3f3700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa6180c1fc0 0x7fa6180c2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:46.252 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.241+0000 7fa61d3f3700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa618067ec0 0x7fa618068340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:46.253 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.241+0000 7fa61d3f3700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa618068880 con 0x7fa604004810 2023-10-06T01:41:46.253 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.241+0000 7fa61d3f3700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6180c32b0 con 0x7fa6180c1fc0 2023-10-06T01:41:46.253 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.241+0000 7fa61d3f3700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6180c6420 con 0x7fa618067ec0 2023-10-06T01:41:46.253 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.241+0000 7fa616ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa604004810 0x7fa61814ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:46.253 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.241+0000 7fa616ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa604004810 0x7fa61814ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39424/0 (socket says 172.21.15.27:39424) 2023-10-06T01:41:46.253 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.241+0000 7fa616ffd700 1 -- 172.21.15.27:0/3610761795 learned_addr learned my addr 172.21.15.27:0/3610761795 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:46.253 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.241+0000 7fa6177fe700 1 --2- 172.21.15.27:0/3610761795 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa618067ec0 0x7fa618068340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:46.254 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.241+0000 7fa6167fc700 1 --2- 172.21.15.27:0/3610761795 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa6180c1fc0 0x7fa6180c2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:46.254 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.242+0000 7fa616ffd700 1 -- 172.21.15.27:0/3610761795 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa618067ec0 msgr2=0x7fa618068340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:46.254 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.242+0000 7fa616ffd700 1 --2- 172.21.15.27:0/3610761795 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa618067ec0 0x7fa618068340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:46.254 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.242+0000 7fa616ffd700 1 -- 172.21.15.27:0/3610761795 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa6180c1fc0 msgr2=0x7fa6180c2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:46.254 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.242+0000 7fa616ffd700 1 --2- 172.21.15.27:0/3610761795 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa6180c1fc0 0x7fa6180c2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:46.254 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.242+0000 7fa616ffd700 1 -- 172.21.15.27:0/3610761795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6180c2a10 con 0x7fa604004810 2023-10-06T01:41:46.254 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.242+0000 7fa6177fe700 1 --2- 172.21.15.27:0/3610761795 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa618067ec0 0x7fa618068340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:46.255 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.242+0000 7fa6167fc700 1 --2- 172.21.15.27:0/3610761795 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa6180c1fc0 0x7fa6180c2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:46.255 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.242+0000 7fa616ffd700 1 --2- 172.21.15.27:0/3610761795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa604004810 0x7fa61814ea20 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa6100073c0 tx=0x7fa610001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:46.255 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.242+0000 7fa60effd700 1 -- 172.21.15.27:0/3610761795 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61000f8d0 con 0x7fa604004810 2023-10-06T01:41:46.255 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.242+0000 7fa60effd700 1 -- 172.21.15.27:0/3610761795 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa61000fa50 con 0x7fa604004810 2023-10-06T01:41:46.255 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.242+0000 7fa60effd700 1 -- 172.21.15.27:0/3610761795 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61000fe10 con 0x7fa604004810 2023-10-06T01:41:46.255 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.242+0000 7fa61d3f3700 1 -- 172.21.15.27:0/3610761795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6180c2cf0 con 0x7fa604004810 2023-10-06T01:41:46.255 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.242+0000 7fa61d3f3700 1 -- 172.21.15.27:0/3610761795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa618153010 con 0x7fa604004810 2023-10-06T01:41:46.255 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.243+0000 7fa61d3f3700 1 -- 172.21.15.27:0/3610761795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6180c61f0 con 0x7fa604004810 2023-10-06T01:41:46.255 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.243+0000 7fa60effd700 1 -- 172.21.15.27:0/3610761795 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa610013600 con 0x7fa604004810 2023-10-06T01:41:46.256 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.244+0000 7fa60effd700 1 --2- 172.21.15.27:0/3610761795 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa5f8066f30 0x7fa5f80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:46.256 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.244+0000 7fa60effd700 1 -- 172.21.15.27:0/3610761795 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fa6100848a0 con 0x7fa604004810 2023-10-06T01:41:46.256 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.244+0000 7fa6167fc700 1 --2- 172.21.15.27:0/3610761795 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa5f8066f30 0x7fa5f80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:46.256 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.245+0000 7fa6167fc700 1 --2- 172.21.15.27:0/3610761795 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa5f8066f30 0x7fa5f80693e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa6000091e0 tx=0x7fa60001a000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:46.256 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.248+0000 7fa60effd700 1 -- 172.21.15.27:0/3610761795 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa6180c61f0 con 0x7fa604004810 2023-10-06T01:41:46.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:46.525+0000 7fa61d3f3700 1 -- 172.21.15.27:0/3610761795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_backtrace.TestBacktrace.test_backtrace"]} v 0) v1 -- 0x7fa6180c61f0 con 0x7fa604004810 2023-10-06T01:41:47.484 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.483+0000 7fa60effd700 1 -- 172.21.15.27:0/3610761795 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_backtrace.TestBacktrace.test_backtrace"]}]=0 v50) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa6180c61f0 con 0x7fa604004810 2023-10-06T01:41:47.488 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.486+0000 7fa61d3f3700 1 -- 172.21.15.27:0/3610761795 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa5f8066f30 msgr2=0x7fa5f80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:47.488 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.486+0000 7fa61d3f3700 1 --2- 172.21.15.27:0/3610761795 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa5f8066f30 0x7fa5f80693e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa6000091e0 tx=0x7fa60001a000 comp rx=0 tx=0).stop 2023-10-06T01:41:47.488 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.486+0000 7fa61d3f3700 1 -- 172.21.15.27:0/3610761795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa604004810 msgr2=0x7fa61814ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:47.488 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.486+0000 7fa61d3f3700 1 --2- 172.21.15.27:0/3610761795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa604004810 0x7fa61814ea20 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fa6100073c0 tx=0x7fa610001d80 comp rx=0 tx=0).stop 2023-10-06T01:41:47.488 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.486+0000 7fa61d3f3700 1 -- 172.21.15.27:0/3610761795 shutdown_connections 2023-10-06T01:41:47.488 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.486+0000 7fa61d3f3700 1 --2- 172.21.15.27:0/3610761795 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa5f8066f30 0x7fa5f80693e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:47.488 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.486+0000 7fa61d3f3700 1 --2- 172.21.15.27:0/3610761795 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa618067ec0 0x7fa618068340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:47.488 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.486+0000 7fa61d3f3700 1 --2- 172.21.15.27:0/3610761795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa604004810 0x7fa61814ea20 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:47.488 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.486+0000 7fa61d3f3700 1 --2- 172.21.15.27:0/3610761795 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa6180c1fc0 0x7fa6180c2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:47.489 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.486+0000 7fa61d3f3700 1 -- 172.21.15.27:0/3610761795 >> 172.21.15.27:0/3610761795 conn(0x7fa61806e520 msgr2=0x7fa618158500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:47.489 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.486+0000 7fa61d3f3700 1 -- 172.21.15.27:0/3610761795 shutdown_connections 2023-10-06T01:41:47.489 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.486+0000 7fa61d3f3700 1 -- 172.21.15.27:0/3610761795 wait complete. 2023-10-06T01:41:47.502 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-06T01:41:47.631 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.630+0000 7f5c75819700 1 Processor -- start 2023-10-06T01:41:47.631 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.630+0000 7f5c75819700 1 -- start start 2023-10-06T01:41:47.633 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.630+0000 7f5c75819700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5c7015d710 0x7f5c700b0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:47.633 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.630+0000 7f5c75819700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5c7015dae0 0x7f5c7015df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:47.633 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.630+0000 7f5c75819700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c700b3960 con 0x7f5c70142b90 2023-10-06T01:41:47.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.630+0000 7f5c75819700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c700cc9b0 con 0x7f5c7015d710 2023-10-06T01:41:47.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.630+0000 7f5c75819700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c700ccc90 con 0x7f5c7015dae0 2023-10-06T01:41:47.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.631+0000 7f5c6effd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5c7015d710 0x7f5c700b0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:47.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.631+0000 7f5c6effd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f5c7015d710 0x7f5c700b0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39434/0 (socket says 172.21.15.27:39434) 2023-10-06T01:41:47.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.631+0000 7f5c6effd700 1 -- 172.21.15.27:0/2395808877 learned_addr learned my addr 172.21.15.27:0/2395808877 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:47.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.631+0000 7f5c6e7fc700 1 --2- 172.21.15.27:0/2395808877 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5c7015dae0 0x7f5c7015df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:47.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.631+0000 7f5c6effd700 1 -- 172.21.15.27:0/2395808877 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5c7015dae0 msgr2=0x7f5c7015df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:47.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.631+0000 7f5c6effd700 1 --2- 172.21.15.27:0/2395808877 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5c7015dae0 0x7f5c7015df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:47.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.631+0000 7f5c6effd700 1 -- 172.21.15.27:0/2395808877 >> v1:172.21.15.124:6789/0 conn(0x7f5c70142b90 legacy=0x7f5c70069330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:47.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.631+0000 7f5c6effd700 1 -- 172.21.15.27:0/2395808877 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c700b3310 con 0x7f5c7015d710 2023-10-06T01:41:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.631+0000 7f5c6effd700 1 --2- 172.21.15.27:0/2395808877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c7015d710 0x7f5c700b0900 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5c600099b0 tx=0x7f5c60009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10c0de4d89823dbd server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.631+0000 7f5c6dffb700 1 -- 172.21.15.27:0/2395808877 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6000b820 con 0x7f5c7015d710 2023-10-06T01:41:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.631+0000 7f5c6dffb700 1 -- 172.21.15.27:0/2395808877 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c6000b980 con 0x7f5c7015d710 2023-10-06T01:41:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.631+0000 7f5c6dffb700 1 -- 172.21.15.27:0/2395808877 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6000bd40 con 0x7f5c7015d710 2023-10-06T01:41:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.632+0000 7f5c75819700 1 -- 172.21.15.27:0/2395808877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c7015d710 msgr2=0x7f5c700b0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.632+0000 7f5c75819700 1 --2- 172.21.15.27:0/2395808877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c7015d710 0x7f5c700b0900 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5c600099b0 tx=0x7f5c60009250 comp rx=0 tx=0).stop 2023-10-06T01:41:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.632+0000 7f5c75819700 1 -- 172.21.15.27:0/2395808877 shutdown_connections 2023-10-06T01:41:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.632+0000 7f5c75819700 1 --2- 172.21.15.27:0/2395808877 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c7015d710 0x7f5c700b0900 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.632+0000 7f5c75819700 1 --2- 172.21.15.27:0/2395808877 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5c7015dae0 0x7f5c7015df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.632+0000 7f5c75819700 1 -- 172.21.15.27:0/2395808877 >> 172.21.15.27:0/2395808877 conn(0x7f5c700d0fd0 msgr2=0x7f5c700ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.632+0000 7f5c75819700 1 -- 172.21.15.27:0/2395808877 shutdown_connections 2023-10-06T01:41:47.635 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.632+0000 7f5c75819700 1 -- 172.21.15.27:0/2395808877 wait complete. 2023-10-06T01:41:47.639 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.633+0000 7f5c75819700 1 Processor -- start 2023-10-06T01:41:47.639 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.633+0000 7f5c75819700 1 -- start start 2023-10-06T01:41:47.639 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.634+0000 7f5c75819700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5c70142b90 0x7f5c7013f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:47.639 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.634+0000 7f5c75819700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c7015d710 0x7f5c700c8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:47.639 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.634+0000 7f5c75819700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5c7015dae0 0x7f5c700c7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:47.639 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.634+0000 7f5c75819700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7013f850 con 0x7f5c7015d710 2023-10-06T01:41:47.639 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.634+0000 7f5c75819700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c700c8930 con 0x7f5c7015dae0 2023-10-06T01:41:47.639 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.634+0000 7f5c75819700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c700c7d20 con 0x7f5c70142b90 2023-10-06T01:41:47.639 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.634+0000 7f5c6e7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c7015d710 0x7f5c700c8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:47.639 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.634+0000 7f5c6e7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c7015d710 0x7f5c700c8dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39438/0 (socket says 172.21.15.27:39438) 2023-10-06T01:41:47.639 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.634+0000 7f5c6e7fc700 1 -- 172.21.15.27:0/239760509 learned_addr learned my addr 172.21.15.27:0/239760509 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:47.640 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.634+0000 7f5c6e7fc700 1 -- 172.21.15.27:0/239760509 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5c70142b90 msgr2=0x7f5c7013f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:47.640 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.634+0000 7f5c6effd700 1 --2- 172.21.15.27:0/239760509 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5c70142b90 0x7f5c7013f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:47.640 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.634+0000 7f5c6e7fc700 1 --2- 172.21.15.27:0/239760509 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5c70142b90 0x7f5c7013f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:47.640 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.634+0000 7f5c6e7fc700 1 -- 172.21.15.27:0/239760509 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5c7015dae0 msgr2=0x7f5c700c7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:47.640 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.634+0000 7f5c6f7fe700 1 --2- 172.21.15.27:0/239760509 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5c7015dae0 0x7f5c700c7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:47.640 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.634+0000 7f5c6e7fc700 1 --2- 172.21.15.27:0/239760509 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5c7015dae0 0x7f5c700c7280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:47.640 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.634+0000 7f5c6e7fc700 1 -- 172.21.15.27:0/239760509 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c600094f0 con 0x7f5c7015d710 2023-10-06T01:41:47.640 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.634+0000 7f5c6e7fc700 1 --2- 172.21.15.27:0/239760509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c7015d710 0x7f5c700c8dc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5c6400c7d0 tx=0x7f5c64008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:47.640 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.634+0000 7f5c56ffd700 1 -- 172.21.15.27:0/239760509 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c64011600 con 0x7f5c7015d710 2023-10-06T01:41:47.640 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.634+0000 7f5c56ffd700 1 -- 172.21.15.27:0/239760509 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c64011760 con 0x7f5c7015d710 2023-10-06T01:41:47.640 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.634+0000 7f5c75819700 1 -- 172.21.15.27:0/239760509 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c700c8000 con 0x7f5c7015d710 2023-10-06T01:41:47.640 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.634+0000 7f5c56ffd700 1 -- 172.21.15.27:0/239760509 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c64011b20 con 0x7f5c7015d710 2023-10-06T01:41:47.641 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.635+0000 7f5c75819700 1 -- 172.21.15.27:0/239760509 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c700c85c0 con 0x7f5c7015d710 2023-10-06T01:41:47.641 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.635+0000 7f5c75819700 1 -- 172.21.15.27:0/239760509 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c700c8aa0 con 0x7f5c7015d710 2023-10-06T01:41:47.641 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.636+0000 7f5c56ffd700 1 -- 172.21.15.27:0/239760509 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f5c640193f0 con 0x7f5c7015d710 2023-10-06T01:41:47.641 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.636+0000 7f5c56ffd700 1 --2- 172.21.15.27:0/239760509 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5c5c066f30 0x7f5c5c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:47.641 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.637+0000 7f5c56ffd700 1 -- 172.21.15.27:0/239760509 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f5c64084aa0 con 0x7f5c7015d710 2023-10-06T01:41:47.641 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.637+0000 7f5c6effd700 1 --2- 172.21.15.27:0/239760509 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5c5c066f30 0x7f5c5c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:47.641 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.637+0000 7f5c6effd700 1 --2- 172.21.15.27:0/239760509 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5c5c066f30 0x7f5c5c0693e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5c600099b0 tx=0x7f5c60016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:47.642 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.641+0000 7f5c56ffd700 1 -- 172.21.15.27:0/239760509 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5c64051bb0 con 0x7f5c7015d710 2023-10-06T01:41:47.834 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.833+0000 7f5c75819700 1 -- 172.21.15.27:0/239760509 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f5c7004c3c0 con 0x7f5c7015d710 2023-10-06T01:41:47.837 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.836+0000 7f5c56ffd700 1 -- 172.21.15.27:0/239760509 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f5c640551d0 con 0x7f5c7015d710 2023-10-06T01:41:47.839 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.838+0000 7f5c75819700 1 -- 172.21.15.27:0/239760509 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5c5c066f30 msgr2=0x7f5c5c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:47.840 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.838+0000 7f5c75819700 1 --2- 172.21.15.27:0/239760509 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5c5c066f30 0x7f5c5c0693e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5c600099b0 tx=0x7f5c60016000 comp rx=0 tx=0).stop 2023-10-06T01:41:47.840 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.838+0000 7f5c75819700 1 -- 172.21.15.27:0/239760509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c7015d710 msgr2=0x7f5c700c8dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:47.840 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.838+0000 7f5c75819700 1 --2- 172.21.15.27:0/239760509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c7015d710 0x7f5c700c8dc0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5c6400c7d0 tx=0x7f5c64008040 comp rx=0 tx=0).stop 2023-10-06T01:41:47.840 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.838+0000 7f5c75819700 1 -- 172.21.15.27:0/239760509 shutdown_connections 2023-10-06T01:41:47.840 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.838+0000 7f5c75819700 1 --2- 172.21.15.27:0/239760509 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5c5c066f30 0x7f5c5c0693e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:47.840 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.838+0000 7f5c75819700 1 --2- 172.21.15.27:0/239760509 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5c70142b90 0x7f5c7013f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:47.840 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.838+0000 7f5c75819700 1 --2- 172.21.15.27:0/239760509 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f5c7015dae0 0x7f5c700c7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:47.840 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.838+0000 7f5c75819700 1 --2- 172.21.15.27:0/239760509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5c7015d710 0x7f5c700c8dc0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:47.840 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.838+0000 7f5c75819700 1 -- 172.21.15.27:0/239760509 >> 172.21.15.27:0/239760509 conn(0x7f5c700d0fd0 msgr2=0x7f5c700ce7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:47.840 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.838+0000 7f5c75819700 1 -- 172.21.15.27:0/239760509 shutdown_connections 2023-10-06T01:41:47.840 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:47.838+0000 7f5c75819700 1 -- 172.21.15.27:0/239760509 wait complete. 2023-10-06T01:41:47.852 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-06T01:41:47.852 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-06T01:41:47.884 DEBUG:teuthology.orchestra.run.smithi124:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-06T01:41:47.931 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-06T01:41:47.931 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-06T01:41:47.931 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T01:41:47.931 DEBUG:teuthology.orchestra.run.smithi027:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2023-10-06T01:41:47.968 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2023-10-06T01:41:47.968 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:47.968 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-06T01:41:48.024 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2023-10-06T01:41:48.024 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T01:41:48.024 DEBUG:teuthology.orchestra.run.smithi124:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2023-10-06T01:41:48.066 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi124.front.sepia.ceph.com 2023-10-06T01:41:48.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:48.066 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-10-06T01:41:48.121 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2023-10-06T01:41:48.121 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T01:41:48.122 DEBUG:teuthology.orchestra.run.smithi027:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2023-10-06T01:41:48.157 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2023-10-06T01:41:48.158 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:48.158 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-10-06T01:41:48.213 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2023-10-06T01:41:48.213 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T01:41:48.213 DEBUG:teuthology.orchestra.run.smithi124:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2023-10-06T01:41:48.247 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi124.front.sepia.ceph.com 2023-10-06T01:41:48.247 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:41:48.247 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-10-06T01:41:48.302 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-06T01:41:48.302 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T01:41:48.421 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.420+0000 7f39cfb07700 1 Processor -- start 2023-10-06T01:41:48.421 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.420+0000 7f39cfb07700 1 -- start start 2023-10-06T01:41:48.423 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.420+0000 7f39cfb07700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f39c8143160 0x7f39c80c43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:48.423 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.420+0000 7f39cfb07700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c8069f50 con 0x7f39c80c4900 2023-10-06T01:41:48.423 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.420+0000 7f39cfb07700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c8069340 con 0x7f39c8142d90 2023-10-06T01:41:48.423 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.420+0000 7f39cfb07700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39c8069620 con 0x7f39c8143160 2023-10-06T01:41:48.423 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.421+0000 7f39cd0a2700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f39c8143160 0x7f39c80c43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:48.423 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.421+0000 7f39cd0a2700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f39c8143160 0x7f39c80c43c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39440/0 (socket says 172.21.15.27:39440) 2023-10-06T01:41:48.423 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.421+0000 7f39cd0a2700 1 -- 172.21.15.27:0/3497700629 learned_addr learned my addr 172.21.15.27:0/3497700629 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:48.424 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.421+0000 7f39ce0a4700 1 --1- 172.21.15.27:0/3497700629 >> v1:172.21.15.27:6789/0 conn(0x7f39c80c4900 0x7f39c80cc130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38442/0 (socket says 172.21.15.27:38442) 2023-10-06T01:41:48.424 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.421+0000 7f39cd0a2700 1 -- 172.21.15.27:0/3497700629 >> v1:172.21.15.124:6789/0 conn(0x7f39c8142d90 legacy=0x7f39c806a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:48.424 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.421+0000 7f39cd0a2700 1 -- 172.21.15.27:0/3497700629 >> v1:172.21.15.27:6789/0 conn(0x7f39c80c4900 legacy=0x7f39c80cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:48.424 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.421+0000 7f39cd0a2700 1 -- 172.21.15.27:0/3497700629 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c8069900 con 0x7f39c8143160 2023-10-06T01:41:48.424 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.421+0000 7f39cc8a1700 1 -- 172.21.15.27:0/3497700629 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 100592318 0 0) 0x7f39c8069f50 con 0x7f39c80c4900 2023-10-06T01:41:48.424 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.421+0000 7f39cd0a2700 1 --2- 172.21.15.27:0/3497700629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39c8143160 0x7f39c80c43c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f39bc0099b0 tx=0x7f39bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97078d07055bbf7c server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:48.424 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.421+0000 7f39cc8a1700 1 -- 172.21.15.27:0/3497700629 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39bc00b860 con 0x7f39c8143160 2023-10-06T01:41:48.424 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.421+0000 7f39cc8a1700 1 -- 172.21.15.27:0/3497700629 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39bc00b9c0 con 0x7f39c8143160 2023-10-06T01:41:48.424 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.421+0000 7f39cc8a1700 1 -- 172.21.15.27:0/3497700629 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39bc00bd60 con 0x7f39c8143160 2023-10-06T01:41:48.424 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.422+0000 7f39cfb07700 1 -- 172.21.15.27:0/3497700629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39c8143160 msgr2=0x7f39c80c43c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:48.424 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.422+0000 7f39cfb07700 1 --2- 172.21.15.27:0/3497700629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39c8143160 0x7f39c80c43c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f39bc0099b0 tx=0x7f39bc009250 comp rx=0 tx=0).stop 2023-10-06T01:41:48.424 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.422+0000 7f39cfb07700 1 -- 172.21.15.27:0/3497700629 shutdown_connections 2023-10-06T01:41:48.425 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.422+0000 7f39cfb07700 1 --2- 172.21.15.27:0/3497700629 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39c8143160 0x7f39c80c43c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:48.425 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.422+0000 7f39cfb07700 1 -- 172.21.15.27:0/3497700629 >> 172.21.15.27:0/3497700629 conn(0x7f39c80ce4d0 msgr2=0x7f39c80ce8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:48.425 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.422+0000 7f39cfb07700 1 -- 172.21.15.27:0/3497700629 shutdown_connections 2023-10-06T01:41:48.425 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.422+0000 7f39cfb07700 1 -- 172.21.15.27:0/3497700629 wait complete. 2023-10-06T01:41:48.427 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.423+0000 7f39cfb07700 1 Processor -- start 2023-10-06T01:41:48.427 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.423+0000 7f39cfb07700 1 -- start start 2023-10-06T01:41:48.427 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.424+0000 7f39cfb07700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f39c80c4900 0x7f39c815ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:48.427 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.424+0000 7f39cfb07700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39c8142d90 0x7f39c8158a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:48.427 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.424+0000 7f39cfb07700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f39c8143160 0x7f39c80c5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:48.427 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.424+0000 7f39cfb07700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c813e3b0 con 0x7f39c8142d90 2023-10-06T01:41:48.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.424+0000 7f39cfb07700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39c815db70 con 0x7f39c8143160 2023-10-06T01:41:48.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.424+0000 7f39cfb07700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39c815de50 con 0x7f39c80c4900 2023-10-06T01:41:48.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.424+0000 7f39cd0a2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39c8142d90 0x7f39c8158a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:48.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.424+0000 7f39cd0a2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39c8142d90 0x7f39c8158a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39446/0 (socket says 172.21.15.27:39446) 2023-10-06T01:41:48.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.424+0000 7f39cd0a2700 1 -- 172.21.15.27:0/3066774434 learned_addr learned my addr 172.21.15.27:0/3066774434 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:48.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.424+0000 7f39ce0a4700 1 --2- 172.21.15.27:0/3066774434 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f39c8143160 0x7f39c80c5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:48.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.424+0000 7f39cd8a3700 1 --2- 172.21.15.27:0/3066774434 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f39c80c4900 0x7f39c815ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:48.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.424+0000 7f39cd0a2700 1 -- 172.21.15.27:0/3066774434 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f39c80c4900 msgr2=0x7f39c815ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:48.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.424+0000 7f39cd0a2700 1 --2- 172.21.15.27:0/3066774434 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f39c80c4900 0x7f39c815ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:48.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.424+0000 7f39cd0a2700 1 -- 172.21.15.27:0/3066774434 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f39c8143160 msgr2=0x7f39c80c5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:48.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.424+0000 7f39cd0a2700 1 --2- 172.21.15.27:0/3066774434 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f39c8143160 0x7f39c80c5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:48.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.424+0000 7f39cd0a2700 1 -- 172.21.15.27:0/3066774434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39bc0094f0 con 0x7f39c8142d90 2023-10-06T01:41:48.428 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.424+0000 7f39cd0a2700 1 --2- 172.21.15.27:0/3066774434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39c8142d90 0x7f39c8158a10 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f39bc006f50 tx=0x7f39bc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:48.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.424+0000 7f39c5ffb700 1 -- 172.21.15.27:0/3066774434 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39bc01f070 con 0x7f39c8142d90 2023-10-06T01:41:48.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.424+0000 7f39c5ffb700 1 -- 172.21.15.27:0/3066774434 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39bc011bb0 con 0x7f39c8142d90 2023-10-06T01:41:48.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.424+0000 7f39c5ffb700 1 -- 172.21.15.27:0/3066774434 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39bc025420 con 0x7f39c8142d90 2023-10-06T01:41:48.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.425+0000 7f39cfb07700 1 -- 172.21.15.27:0/3066774434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39c815e130 con 0x7f39c8142d90 2023-10-06T01:41:48.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.425+0000 7f39cfb07700 1 -- 172.21.15.27:0/3066774434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39c815e6f0 con 0x7f39c8142d90 2023-10-06T01:41:48.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.426+0000 7f39cfb07700 1 -- 172.21.15.27:0/3066774434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39c8064050 con 0x7f39c8142d90 2023-10-06T01:41:48.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.426+0000 7f39c5ffb700 1 -- 172.21.15.27:0/3066774434 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f39bc023070 con 0x7f39c8142d90 2023-10-06T01:41:48.430 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.426+0000 7f39c5ffb700 1 --2- 172.21.15.27:0/3066774434 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f39b4066f30 0x7f39b40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:48.430 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.426+0000 7f39c5ffb700 1 -- 172.21.15.27:0/3066774434 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f39bc085750 con 0x7f39c8142d90 2023-10-06T01:41:48.430 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.427+0000 7f39cd8a3700 1 --2- 172.21.15.27:0/3066774434 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f39b4066f30 0x7f39b40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:48.430 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.427+0000 7f39cd8a3700 1 --2- 172.21.15.27:0/3066774434 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f39b4066f30 0x7f39b40693e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f39c815f370 tx=0x7f39b8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:48.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.431+0000 7f39c5ffb700 1 -- 172.21.15.27:0/3066774434 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f39bc052860 con 0x7f39c8142d90 2023-10-06T01:41:48.573 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T01:41:48.575 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T01:41:48.670 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.669+0000 7f39cfb07700 1 -- 172.21.15.27:0/3066774434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39c815e8e0 con 0x7f39c8142d90 2023-10-06T01:41:48.671 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.670+0000 7f39c5ffb700 1 -- 172.21.15.27:0/3066774434 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4594 (secure 0 0 0) 0x7f39bc055e80 con 0x7f39c8142d90 2023-10-06T01:41:48.671 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:48.671 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4261,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6837/2699906167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6836","nonce":2699906167},{"type":"v1","addr":"172.21.15.124:6837","nonce":2699906167}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/313372271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":313372271},{"type":"v1","addr":"172.21.15.27:6834","nonce":313372271}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4287,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/4257743842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4257743842},{"type":"v1","addr":"172.21.15.27:6835","nonce":4257743842}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T01:41:12.735970+0000","modified":"2023-10-06T01:41:16.358924+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.124:6835/3803994994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":3803994994},{"type":"v1","addr":"172.21.15.124:6835","nonce":3803994994}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-06T01:41:48.673 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.672+0000 7f39cfb07700 1 -- 172.21.15.27:0/3066774434 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f39b4066f30 msgr2=0x7f39b40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:48.674 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.672+0000 7f39cfb07700 1 --2- 172.21.15.27:0/3066774434 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f39b4066f30 0x7f39b40693e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f39c815f370 tx=0x7f39b8009040 comp rx=0 tx=0).stop 2023-10-06T01:41:48.674 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.672+0000 7f39cfb07700 1 -- 172.21.15.27:0/3066774434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39c8142d90 msgr2=0x7f39c8158a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:48.674 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.672+0000 7f39cfb07700 1 --2- 172.21.15.27:0/3066774434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39c8142d90 0x7f39c8158a10 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f39bc006f50 tx=0x7f39bc017000 comp rx=0 tx=0).stop 2023-10-06T01:41:48.674 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.672+0000 7f39cfb07700 1 -- 172.21.15.27:0/3066774434 shutdown_connections 2023-10-06T01:41:48.674 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.672+0000 7f39cfb07700 1 --2- 172.21.15.27:0/3066774434 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f39b4066f30 0x7f39b40693e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:48.674 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.672+0000 7f39cfb07700 1 --2- 172.21.15.27:0/3066774434 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f39c80c4900 0x7f39c815ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:48.674 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.672+0000 7f39cfb07700 1 --2- 172.21.15.27:0/3066774434 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f39c8143160 0x7f39c80c5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:48.674 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.672+0000 7f39cfb07700 1 --2- 172.21.15.27:0/3066774434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39c8142d90 0x7f39c8158a10 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:48.674 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.672+0000 7f39cfb07700 1 -- 172.21.15.27:0/3066774434 >> 172.21.15.27:0/3066774434 conn(0x7f39c80ce4d0 msgr2=0x7f39c806c8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:48.674 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.672+0000 7f39cfb07700 1 -- 172.21.15.27:0/3066774434 shutdown_connections 2023-10-06T01:41:48.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.672+0000 7f39cfb07700 1 -- 172.21.15.27:0/3066774434 wait complete. 2023-10-06T01:41:48.675 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 7 2023-10-06T01:41:48.685 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T01:41:48.804 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.803+0000 7f30c8f67700 1 Processor -- start 2023-10-06T01:41:48.804 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.804+0000 7f30c8f67700 1 -- start start 2023-10-06T01:41:48.808 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.804+0000 7f30c8f67700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f30c4157570 0x7f30c40b4060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:48.808 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.804+0000 7f30c8f67700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f30c40add70 0x7f30c40ae190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:48.808 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.804+0000 7f30c8f67700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c41580d0 con 0x7f30c415b210 2023-10-06T01:41:48.808 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.804+0000 7f30c8f67700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30c4158c70 con 0x7f30c4157570 2023-10-06T01:41:48.808 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.804+0000 7f30c8f67700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30c40ae6d0 con 0x7f30c40add70 2023-10-06T01:41:48.808 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.805+0000 7f30c1d9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f30c415b210 0x7f30c4158560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38456/0 (socket says 172.21.15.27:38456) 2023-10-06T01:41:48.809 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.805+0000 7f30c1d9b700 1 -- 172.21.15.27:0/2072315707 learned_addr learned my addr 172.21.15.27:0/2072315707 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:48.809 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.805+0000 7f30c259c700 1 --2- 172.21.15.27:0/2072315707 >> v2:172.21.15.124:3300/0 conn(0x7f30c4157570 0x7f30c40b4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:48.809 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.805+0000 7f30c2d9d700 1 --2- 172.21.15.27:0/2072315707 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f30c40add70 0x7f30c40ae190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:48.809 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.805+0000 7f30c159a700 1 -- 172.21.15.27:0/2072315707 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2572078708 0 0) 0x7f30c41580d0 con 0x7f30c415b210 2023-10-06T01:41:48.809 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.805+0000 7f30c159a700 1 -- 172.21.15.27:0/2072315707 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30b0003570 con 0x7f30c415b210 2023-10-06T01:41:48.809 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.805+0000 7f30c159a700 1 -- 172.21.15.27:0/2072315707 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3644581889 0 0) 0x7f30b0003570 con 0x7f30c415b210 2023-10-06T01:41:48.809 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.805+0000 7f30c159a700 1 -- 172.21.15.27:0/2072315707 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f30c40add70 msgr2=0x7f30c40ae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:48.809 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.805+0000 7f30c159a700 1 --2- 172.21.15.27:0/2072315707 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f30c40add70 0x7f30c40ae190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:48.809 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.805+0000 7f30c159a700 1 -- 172.21.15.27:0/2072315707 >> v2:172.21.15.124:3300/0 conn(0x7f30c4157570 msgr2=0x7f30c40b4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:48.809 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.805+0000 7f30c159a700 1 --2- 172.21.15.27:0/2072315707 >> v2:172.21.15.124:3300/0 conn(0x7f30c4157570 0x7f30c40b4060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:48.809 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.805+0000 7f30c159a700 1 -- 172.21.15.27:0/2072315707 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c415ba40 con 0x7f30c415b210 2023-10-06T01:41:48.809 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.805+0000 7f30c159a700 1 -- 172.21.15.27:0/2072315707 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2619078944 0 0) 0x7f30ac001470 con 0x7f30c415b210 2023-10-06T01:41:48.809 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.805+0000 7f30c159a700 1 -- 172.21.15.27:0/2072315707 >> v1:172.21.15.27:6789/0 conn(0x7f30c415b210 legacy=0x7f30c4158560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:48.810 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.805+0000 7f30c159a700 1 --2- 172.21.15.27:0/2072315707 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f30c40add70 0x7f30b0003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:48.810 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.805+0000 7f30c159a700 1 --2- 172.21.15.27:0/2072315707 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f30b0003fe0 0x7f30b0004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:48.810 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.805+0000 7f30c159a700 1 --2- 172.21.15.27:0/2072315707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30b0004d50 0x7f30b00051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:48.810 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.805+0000 7f30c159a700 1 -- 172.21.15.27:0/2072315707 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c4158c70 con 0x7f30b0004d50 2023-10-06T01:41:48.810 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.805+0000 7f30c159a700 1 -- 172.21.15.27:0/2072315707 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c40ae6d0 con 0x7f30b0003fe0 2023-10-06T01:41:48.810 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.805+0000 7f30c159a700 1 -- 172.21.15.27:0/2072315707 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30b000ba90 con 0x7f30c40add70 2023-10-06T01:41:48.810 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.805+0000 7f30c2d9d700 1 --2- 172.21.15.27:0/2072315707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30b0004d50 0x7f30b00051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:48.810 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.806+0000 7f30c259c700 1 --2- 172.21.15.27:0/2072315707 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f30c40add70 0x7f30b0003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:48.810 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.806+0000 7f30c1d9b700 1 --2- 172.21.15.27:0/2072315707 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f30b0003fe0 0x7f30b0004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:48.810 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.806+0000 7f30c2d9d700 1 -- 172.21.15.27:0/2072315707 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f30c40add70 msgr2=0x7f30b0003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:48.810 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.806+0000 7f30c2d9d700 1 --2- 172.21.15.27:0/2072315707 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f30c40add70 0x7f30b0003a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:48.811 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.806+0000 7f30c2d9d700 1 -- 172.21.15.27:0/2072315707 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f30b0003fe0 msgr2=0x7f30b0004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:48.811 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.806+0000 7f30c2d9d700 1 --2- 172.21.15.27:0/2072315707 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f30b0003fe0 0x7f30b0004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:48.811 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.806+0000 7f30c2d9d700 1 -- 172.21.15.27:0/2072315707 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b0003570 con 0x7f30b0004d50 2023-10-06T01:41:48.811 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.806+0000 7f30c2d9d700 1 --2- 172.21.15.27:0/2072315707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30b0004d50 0x7f30b00051c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f30b400e210 tx=0x7f30b400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b147c5e67906eb2a server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:48.811 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.806+0000 7f30c159a700 1 -- 172.21.15.27:0/2072315707 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b400ea60 con 0x7f30b0004d50 2023-10-06T01:41:48.811 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.806+0000 7f30c159a700 1 -- 172.21.15.27:0/2072315707 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30b400f040 con 0x7f30b0004d50 2023-10-06T01:41:48.811 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.806+0000 7f30c159a700 1 -- 172.21.15.27:0/2072315707 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b400ee10 con 0x7f30b0004d50 2023-10-06T01:41:48.811 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.807+0000 7f30c8f67700 1 -- 172.21.15.27:0/2072315707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30b0004d50 msgr2=0x7f30b00051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:48.811 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.807+0000 7f30c8f67700 1 --2- 172.21.15.27:0/2072315707 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30b0004d50 0x7f30b00051c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f30b400e210 tx=0x7f30b400b040 comp rx=0 tx=0).stop 2023-10-06T01:41:48.811 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.807+0000 7f30c2d9d700 1 -- 172.21.15.27:0/2072315707 reap_dead start 2023-10-06T01:41:48.811 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.807+0000 7f30c8f67700 1 -- 172.21.15.27:0/2072315707 shutdown_connections 2023-10-06T01:41:48.811 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.807+0000 7f30c8f67700 1 -- 172.21.15.27:0/2072315707 >> 172.21.15.27:0/2072315707 conn(0x7f30c4067630 msgr2=0x7f30c4067a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:48.812 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.807+0000 7f30c8f67700 1 -- 172.21.15.27:0/2072315707 shutdown_connections 2023-10-06T01:41:48.812 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.807+0000 7f30c8f67700 1 -- 172.21.15.27:0/2072315707 wait complete. 2023-10-06T01:41:48.812 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.808+0000 7f30c8f67700 1 Processor -- start 2023-10-06T01:41:48.812 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.808+0000 7f30c8f67700 1 -- start start 2023-10-06T01:41:48.814 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.809+0000 7f30c8f67700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f30b0004d50 0x7f30c4155010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:48.814 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.809+0000 7f30c8f67700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30c40c4a10 0x7f30c40c99b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:48.814 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.809+0000 7f30c8f67700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f30c40b1760 0x7f30c40b1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:48.815 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.809+0000 7f30c8f67700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c40b2110 con 0x7f30c40c4a10 2023-10-06T01:41:48.815 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.809+0000 7f30c8f67700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c40ca270 con 0x7f30c40b1760 2023-10-06T01:41:48.815 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.809+0000 7f30c8f67700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30c40ca550 con 0x7f30b0004d50 2023-10-06T01:41:48.815 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.809+0000 7f30c1d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30c40c4a10 0x7f30c40c99b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:48.815 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.809+0000 7f30c1d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30c40c4a10 0x7f30c40c99b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39472/0 (socket says 172.21.15.27:39472) 2023-10-06T01:41:48.815 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.809+0000 7f30c1d9b700 1 -- 172.21.15.27:0/2551910266 learned_addr learned my addr 172.21.15.27:0/2551910266 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:48.815 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.809+0000 7f30c259c700 1 --2- 172.21.15.27:0/2551910266 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f30b0004d50 0x7f30c4155010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:48.815 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.809+0000 7f30c2d9d700 1 --2- 172.21.15.27:0/2551910266 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f30c40b1760 0x7f30c40b1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:48.815 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.809+0000 7f30c1d9b700 1 -- 172.21.15.27:0/2551910266 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f30b0004d50 msgr2=0x7f30c4155010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:48.815 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.809+0000 7f30c1d9b700 1 --2- 172.21.15.27:0/2551910266 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f30b0004d50 0x7f30c4155010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:48.815 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.809+0000 7f30c1d9b700 1 -- 172.21.15.27:0/2551910266 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f30c40b1760 msgr2=0x7f30c40b1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:48.816 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.809+0000 7f30c1d9b700 1 --2- 172.21.15.27:0/2551910266 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f30c40b1760 0x7f30c40b1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:48.816 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.809+0000 7f30c1d9b700 1 -- 172.21.15.27:0/2551910266 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c40ca830 con 0x7f30c40c4a10 2023-10-06T01:41:48.816 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.809+0000 7f30c1d9b700 1 --2- 172.21.15.27:0/2551910266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30c40c4a10 0x7f30c40c99b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f30ac000fd0 tx=0x7f30ac001e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:48.816 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.809+0000 7f30ba7fc700 1 -- 172.21.15.27:0/2551910266 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ac00f8d0 con 0x7f30c40c4a10 2023-10-06T01:41:48.816 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.809+0000 7f30ba7fc700 1 -- 172.21.15.27:0/2551910266 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30ac00fa50 con 0x7f30c40c4a10 2023-10-06T01:41:48.816 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.810+0000 7f30c8f67700 1 -- 172.21.15.27:0/2551910266 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30c40cab10 con 0x7f30c40c4a10 2023-10-06T01:41:48.816 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.810+0000 7f30ba7fc700 1 -- 172.21.15.27:0/2551910266 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ac00fe10 con 0x7f30c40c4a10 2023-10-06T01:41:48.816 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.810+0000 7f30c8f67700 1 -- 172.21.15.27:0/2551910266 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c4155550 con 0x7f30c40c4a10 2023-10-06T01:41:48.816 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.810+0000 7f30c8f67700 1 -- 172.21.15.27:0/2551910266 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30c40c69f0 con 0x7f30c40c4a10 2023-10-06T01:41:48.816 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.811+0000 7f30ba7fc700 1 -- 172.21.15.27:0/2551910266 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f30ac013600 con 0x7f30c40c4a10 2023-10-06T01:41:48.816 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.811+0000 7f30ba7fc700 1 --2- 172.21.15.27:0/2551910266 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f30a4066f30 0x7f30a40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:48.816 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.811+0000 7f30ba7fc700 1 -- 172.21.15.27:0/2551910266 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f30ac0848a0 con 0x7f30c40c4a10 2023-10-06T01:41:48.817 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.812+0000 7f30c259c700 1 --2- 172.21.15.27:0/2551910266 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f30a4066f30 0x7f30a40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:48.817 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.812+0000 7f30c259c700 1 --2- 172.21.15.27:0/2551910266 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f30a4066f30 0x7f30a40693e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f30bc006f00 tx=0x7f30bc0091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:48.817 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:48.816+0000 7f30ba7fc700 1 -- 172.21.15.27:0/2551910266 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f30c40c69f0 con 0x7f30c40c4a10 2023-10-06T01:41:49.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.011+0000 7f30c8f67700 1 -- 172.21.15.27:0/2551910266 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30c40c69f0 con 0x7f30c40c4a10 2023-10-06T01:41:49.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.012+0000 7f30ba7fc700 1 -- 172.21.15.27:0/2551910266 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16866 (secure 0 0 0) 0x7f30c40c69f0 con 0x7f30c40c4a10 2023-10-06T01:41:49.013 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:49.013 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":20,"fsid":"20a7bff0-74f8-4f2f-b6c6-899e804bc965","created":"2023-10-06T01:40:54.474215+0000","modified":"2023-10-06T01:41:14.324546+0000","last_up_change":"2023-10-06T01:41:05.242847+0000","last_in_change":"2023-10-06T01:40:56.890960+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-10-06T01:41:05.685386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-10-06T01:41:06.092995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T01:41:10.434257+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T01:41:11.700262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b902c86d-800c-40c5-b152-b545e50d02c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":90644},{"type":"v1","addr":"172.21.15.27:6801","nonce":90644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":90644},{"type":"v1","addr":"172.21.15.27:6803","nonce":90644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":90644},{"type":"v1","addr":"172.21.15.27:6807","nonce":90644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":90644},{"type":"v1","addr":"172.21.15.27:6805","nonce":90644}]},"public_addr":"172.21.15.27:6801/90644","cluster_addr":"172.21.15.27:6803/90644","heartbeat_back_addr":"172.21.15.27:6807/90644","heartbeat_front_addr":"172.21.15.27:6805/90644","state":["exists","up"]},{"osd":1,"uuid":"5b4a70e5-b06e-48cd-bfc4-7962c5d05386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":90643},{"type":"v1","addr":"172.21.15.27:6809","nonce":90643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":90643},{"type":"v1","addr":"172.21.15.27:6811","nonce":90643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":90643},{"type":"v1","addr":"172.21.15.27:6815","nonce":90643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":90643},{"type":"v1","addr":"172.21.15.27:6813","nonce":90643}]},"public_addr":"172.21.15.27:6809/90643","cluster_addr":"172.21.15.27:6811/90643","heartbeat_back_addr":"172.21.15.27:6815/90643","heartbeat_front_addr":"172.21.15.27:6813/90643","state":["exists","up"]},{"osd":2,"uuid":"22d9de46-4073-4336-9122-5227e601eca0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":90645},{"type":"v1","addr":"172.21.15.27:6817","nonce":90645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":90645},{"type":"v1","addr":"172.21.15.27:6819","nonce":90645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":90645},{"type":"v1","addr":"172.21.15.27:6823","nonce":90645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":90645},{"type":"v1","addr":"172.21.15.27:6821","nonce":90645}]},"public_addr":"172.21.15.27:6817/90645","cluster_addr":"172.21.15.27:6819/90645","heartbeat_back_addr":"172.21.15.27:6823/90645","heartbeat_front_addr":"172.21.15.27:6821/90645","state":["exists","up"]},{"osd":3,"uuid":"438bb2d3-34e2-4b7c-9734-8360c1b67650","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":90668},{"type":"v1","addr":"172.21.15.27:6825","nonce":90668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":90668},{"type":"v1","addr":"172.21.15.27:6827","nonce":90668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":90668},{"type":"v1","addr":"172.21.15.27:6831","nonce":90668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":90668},{"type":"v1","addr":"172.21.15.27:6829","nonce":90668}]},"public_addr":"172.21.15.27:6825/90668","cluster_addr":"172.21.15.27:6827/90668","heartbeat_back_addr":"172.21.15.27:6831/90668","heartbeat_front_addr":"172.21.15.27:6829/90668","state":["exists","up"]},{"osd":4,"uuid":"e1956a8e-e0f8-4e7c-acbb-40037415a354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":90863},{"type":"v1","addr":"172.21.15.124:6801","nonce":90863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":90863},{"type":"v1","addr":"172.21.15.124:6803","nonce":90863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":90863},{"type":"v1","addr":"172.21.15.124:6807","nonce":90863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":90863},{"type":"v1","addr":"172.21.15.124:6805","nonce":90863}]},"public_addr":"172.21.15.124:6801/90863","cluster_addr":"172.21.15.124:6803/90863","heartbeat_back_addr":"172.21.15.124:6807/90863","heartbeat_front_addr":"172.21.15.124:6805/90863","state":["exists","up"]},{"osd":5,"uuid":"306c59d4-846d-4b73-b1c9-051bec9e9368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":90885},{"type":"v1","addr":"172.21.15.124:6817","nonce":90885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":90885},{"type":"v1","addr":"172.21.15.124:6819","nonce":90885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":90885},{"type":"v1","addr":"172.21.15.124:6823","nonce":90885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":90885},{"type":"v1","addr":"172.21.15.124:6821","nonce":90885}]},"public_addr":"172.21.15.124:6817/90885","cluster_addr":"172.21.15.124:6819/90885","heartbeat_back_addr":"172.21.15.124:6823/90885","heartbeat_front_addr":"172.21.15.124:6821/90885","state":["exists","up"]},{"osd":6,"uuid":"15463e53-ac8d-47d1-b4f6-c0d31f29e4a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":90864},{"type":"v1","addr":"172.21.15.124:6809","nonce":90864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":90864},{"type":"v1","addr":"172.21.15.124:6811","nonce":90864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":90864},{"type":"v1","addr":"172.21.15.124:6815","nonce":90864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":90864},{"type":"v1","addr":"172.21.15.124:6813","nonce":90864}]},"public_addr":"172.21.15.124:6809/90864","cluster_addr":"172.21.15.124:6811/90864","heartbeat_back_addr":"172.21.15.124:6815/90864","heartbeat_front_addr":"172.21.15.124:6813/90864","state":["exists","up"]},{"osd":7,"uuid":"ae463a90-a9d3-498f-86fa-47a35b3fe889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":90894},{"type":"v1","addr":"172.21.15.124:6825","nonce":90894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":90894},{"type":"v1","addr":"172.21.15.124:6827","nonce":90894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":90894},{"type":"v1","addr":"172.21.15.124:6831","nonce":90894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":90894},{"type":"v1","addr":"172.21.15.124:6829","nonce":90894}]},"public_addr":"172.21.15.124:6825/90894","cluster_addr":"172.21.15.124:6827/90894","heartbeat_back_addr":"172.21.15.124:6831/90894","heartbeat_front_addr":"172.21.15.124:6829/90894","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.827704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.839951+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.032230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.050605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.987908+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.031290+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.080707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.053692+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T01:41:49.015 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.014+0000 7f30c8f67700 1 -- 172.21.15.27:0/2551910266 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f30a4066f30 msgr2=0x7f30a40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:49.015 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.014+0000 7f30c8f67700 1 --2- 172.21.15.27:0/2551910266 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f30a4066f30 0x7f30a40693e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f30bc006f00 tx=0x7f30bc0091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:49.015 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.014+0000 7f30c8f67700 1 -- 172.21.15.27:0/2551910266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30c40c4a10 msgr2=0x7f30c40c99b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:49.015 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.014+0000 7f30c8f67700 1 --2- 172.21.15.27:0/2551910266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30c40c4a10 0x7f30c40c99b0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f30ac000fd0 tx=0x7f30ac001e90 comp rx=0 tx=0).stop 2023-10-06T01:41:49.015 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.014+0000 7f30c8f67700 1 -- 172.21.15.27:0/2551910266 shutdown_connections 2023-10-06T01:41:49.015 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.014+0000 7f30c8f67700 1 --2- 172.21.15.27:0/2551910266 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f30a4066f30 0x7f30a40693e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:49.015 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.014+0000 7f30c8f67700 1 --2- 172.21.15.27:0/2551910266 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f30b0004d50 0x7f30c4155010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:49.015 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.014+0000 7f30c8f67700 1 --2- 172.21.15.27:0/2551910266 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f30c40b1760 0x7f30c40b1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:49.016 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.014+0000 7f30c8f67700 1 --2- 172.21.15.27:0/2551910266 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30c40c4a10 0x7f30c40c99b0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:49.016 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.014+0000 7f30c8f67700 1 -- 172.21.15.27:0/2551910266 >> 172.21.15.27:0/2551910266 conn(0x7f30c4067630 msgr2=0x7f30c406c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:49.016 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.014+0000 7f30c8f67700 1 -- 172.21.15.27:0/2551910266 shutdown_connections 2023-10-06T01:41:49.016 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.014+0000 7f30c8f67700 1 -- 172.21.15.27:0/2551910266 wait complete. 2023-10-06T01:41:49.027 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-10-06T01:41:05.685386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-10-06T01:41:06.092995+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-06T01:41:10.434257+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2023-10-06T01:41:11.700262+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-06T01:41:49.028 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-06T01:41:49.150 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.150+0000 7f805aa54700 1 Processor -- start 2023-10-06T01:41:49.150 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.150+0000 7f805aa54700 1 -- start start 2023-10-06T01:41:49.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.151+0000 7f805aa54700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f805415dab0 0x7f80540cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:49.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.151+0000 7f805aa54700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80540b1110 con 0x7f8054159500 2023-10-06T01:41:49.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.151+0000 7f805aa54700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80540b13b0 con 0x7f80541598d0 2023-10-06T01:41:49.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.151+0000 7f805aa54700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80540b1690 con 0x7f805415dab0 2023-10-06T01:41:49.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.151+0000 7f8053fff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f8054159500 0x7f80540b3fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38460/0 (socket says 172.21.15.27:38460) 2023-10-06T01:41:49.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.151+0000 7f8053fff700 1 -- 172.21.15.27:0/3067925795 learned_addr learned my addr 172.21.15.27:0/3067925795 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:49.158 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.151+0000 7f8058ff1700 1 --2- 172.21.15.27:0/3067925795 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f805415dab0 0x7f80540cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:49.158 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.151+0000 7f8052ffd700 1 -- 172.21.15.27:0/3067925795 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3464212848 0 0) 0x7f80540b1110 con 0x7f8054159500 2023-10-06T01:41:49.158 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.151+0000 7f8052ffd700 1 -- 172.21.15.27:0/3067925795 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f803c003570 con 0x7f8054159500 2023-10-06T01:41:49.158 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.151+0000 7f8052ffd700 1 -- 172.21.15.27:0/3067925795 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2307747274 0 0) 0x7f80540b13b0 con 0x7f80541598d0 2023-10-06T01:41:49.158 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.151+0000 7f8052ffd700 1 -- 172.21.15.27:0/3067925795 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80540b1110 con 0x7f80541598d0 2023-10-06T01:41:49.158 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.151+0000 7f8052ffd700 1 -- 172.21.15.27:0/3067925795 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1592603323 0 0) 0x7f803c003570 con 0x7f8054159500 2023-10-06T01:41:49.158 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.151+0000 7f8052ffd700 1 -- 172.21.15.27:0/3067925795 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f805415dab0 msgr2=0x7f80540cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:49.158 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.151+0000 7f8052ffd700 1 --2- 172.21.15.27:0/3067925795 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f805415dab0 0x7f80540cf6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:49.159 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.151+0000 7f8052ffd700 1 -- 172.21.15.27:0/3067925795 >> v1:172.21.15.124:6789/0 conn(0x7f80541598d0 legacy=0x7f805415d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:49.159 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.151+0000 7f8052ffd700 1 -- 172.21.15.27:0/3067925795 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80540b1970 con 0x7f8054159500 2023-10-06T01:41:49.159 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.151+0000 7f8052ffd700 1 -- 172.21.15.27:0/3067925795 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2619078944 0 0) 0x7f8044001470 con 0x7f8054159500 2023-10-06T01:41:49.159 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.151+0000 7f8052ffd700 1 -- 172.21.15.27:0/3067925795 >> v1:172.21.15.27:6789/0 conn(0x7f8054159500 legacy=0x7f80540b3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:49.159 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.151+0000 7f8052ffd700 1 --2- 172.21.15.27:0/3067925795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f803c003c20 0x7f803c004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:49.159 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.151+0000 7f8052ffd700 1 --2- 172.21.15.27:0/3067925795 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f803c004810 0x7f803c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:49.159 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.151+0000 7f8052ffd700 1 --2- 172.21.15.27:0/3067925795 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f805415dab0 0x7f803c005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:49.159 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.152+0000 7f8052ffd700 1 -- 172.21.15.27:0/3067925795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80540b1690 con 0x7f803c003c20 2023-10-06T01:41:49.159 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.152+0000 7f8052ffd700 1 -- 172.21.15.27:0/3067925795 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f803c00bbf0 con 0x7f803c004810 2023-10-06T01:41:49.159 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.152+0000 7f8052ffd700 1 -- 172.21.15.27:0/3067925795 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f803c00bdb0 con 0x7f805415dab0 2023-10-06T01:41:49.160 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.152+0000 7f8053fff700 1 --2- 172.21.15.27:0/3067925795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f803c003c20 0x7f803c004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:49.160 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.152+0000 7f80537fe700 1 --2- 172.21.15.27:0/3067925795 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f803c004810 0x7f803c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:49.160 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.152+0000 7f8058ff1700 1 --2- 172.21.15.27:0/3067925795 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f805415dab0 0x7f803c005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:49.160 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.152+0000 7f8053fff700 1 -- 172.21.15.27:0/3067925795 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f805415dab0 msgr2=0x7f803c005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:49.160 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.152+0000 7f8053fff700 1 --2- 172.21.15.27:0/3067925795 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f805415dab0 0x7f803c005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:49.160 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.152+0000 7f8053fff700 1 -- 172.21.15.27:0/3067925795 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f803c004810 msgr2=0x7f803c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:49.160 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.152+0000 7f8053fff700 1 --2- 172.21.15.27:0/3067925795 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f803c004810 0x7f803c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:49.160 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.152+0000 7f8053fff700 1 -- 172.21.15.27:0/3067925795 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f803c003570 con 0x7f803c003c20 2023-10-06T01:41:49.160 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.152+0000 7f8053fff700 1 --2- 172.21.15.27:0/3067925795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f803c003c20 0x7f803c004010 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f804400c1e0 tx=0x7f805415e380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=182d6011b54755a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:49.160 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.152+0000 7f8052ffd700 1 -- 172.21.15.27:0/3067925795 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804400cb10 con 0x7f803c003c20 2023-10-06T01:41:49.160 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.152+0000 7f8052ffd700 1 -- 172.21.15.27:0/3067925795 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f804400cc90 con 0x7f803c003c20 2023-10-06T01:41:49.161 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.152+0000 7f8052ffd700 1 -- 172.21.15.27:0/3067925795 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8044005b70 con 0x7f803c003c20 2023-10-06T01:41:49.161 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.153+0000 7f805aa54700 1 -- 172.21.15.27:0/3067925795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f803c003c20 msgr2=0x7f803c004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:49.161 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.153+0000 7f805aa54700 1 --2- 172.21.15.27:0/3067925795 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f803c003c20 0x7f803c004010 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f804400c1e0 tx=0x7f805415e380 comp rx=0 tx=0).stop 2023-10-06T01:41:49.161 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.153+0000 7f8058ff1700 1 -- 172.21.15.27:0/3067925795 reap_dead start 2023-10-06T01:41:49.161 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.153+0000 7f805aa54700 1 -- 172.21.15.27:0/3067925795 shutdown_connections 2023-10-06T01:41:49.161 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.153+0000 7f805aa54700 1 -- 172.21.15.27:0/3067925795 >> 172.21.15.27:0/3067925795 conn(0x7f8054067630 msgr2=0x7f8054069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:49.161 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.153+0000 7f805aa54700 1 -- 172.21.15.27:0/3067925795 shutdown_connections 2023-10-06T01:41:49.161 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.153+0000 7f805aa54700 1 -- 172.21.15.27:0/3067925795 wait complete. 2023-10-06T01:41:49.161 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.154+0000 7f805aa54700 1 Processor -- start 2023-10-06T01:41:49.161 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.154+0000 7f805aa54700 1 -- start start 2023-10-06T01:41:49.161 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.155+0000 7f805aa54700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f803c003c20 0x7f8054141fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:49.161 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.155+0000 7f805aa54700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f80540ceb80 0x7f80540cefb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:49.162 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.155+0000 7f805aa54700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8054150a30 0x7f8054150ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:49.162 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.155+0000 7f805aa54700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8054152280 con 0x7f8054150a30 2023-10-06T01:41:49.162 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.155+0000 7f805aa54700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f805415d630 con 0x7f803c003c20 2023-10-06T01:41:49.162 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.155+0000 7f805aa54700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8054151410 con 0x7f80540ceb80 2023-10-06T01:41:49.162 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.155+0000 7f8058ff1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8054150a30 0x7f8054150ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:49.162 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.155+0000 7f8058ff1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8054150a30 0x7f8054150ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39490/0 (socket says 172.21.15.27:39490) 2023-10-06T01:41:49.162 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.155+0000 7f8058ff1700 1 -- 172.21.15.27:0/478789963 learned_addr learned my addr 172.21.15.27:0/478789963 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:49.162 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.155+0000 7f80537fe700 1 --2- 172.21.15.27:0/478789963 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f80540ceb80 0x7f80540cefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:49.162 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.155+0000 7f8053fff700 1 --2- 172.21.15.27:0/478789963 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f803c003c20 0x7f8054141fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:49.163 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.155+0000 7f8058ff1700 1 -- 172.21.15.27:0/478789963 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f80540ceb80 msgr2=0x7f80540cefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:49.163 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.155+0000 7f8058ff1700 1 --2- 172.21.15.27:0/478789963 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f80540ceb80 0x7f80540cefb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:49.163 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.155+0000 7f8058ff1700 1 -- 172.21.15.27:0/478789963 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f803c003c20 msgr2=0x7f8054141fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:49.163 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.155+0000 7f8058ff1700 1 --2- 172.21.15.27:0/478789963 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f803c003c20 0x7f8054141fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:49.163 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.155+0000 7f8058ff1700 1 -- 172.21.15.27:0/478789963 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80540cf4f0 con 0x7f8054150a30 2023-10-06T01:41:49.163 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.155+0000 7f80537fe700 1 --2- 172.21.15.27:0/478789963 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f80540ceb80 0x7f80540cefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:41:49.163 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.155+0000 7f8053fff700 1 --2- 172.21.15.27:0/478789963 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f803c003c20 0x7f8054141fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:41:49.163 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.155+0000 7f8058ff1700 1 --2- 172.21.15.27:0/478789963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8054150a30 0x7f8054150ed0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f804c0094d0 tx=0x7f804c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:49.163 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.156+0000 7f803bfff700 1 -- 172.21.15.27:0/478789963 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804c00e040 con 0x7f8054150a30 2023-10-06T01:41:49.164 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.156+0000 7f803bfff700 1 -- 172.21.15.27:0/478789963 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f804c00f040 con 0x7f8054150a30 2023-10-06T01:41:49.164 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.156+0000 7f805aa54700 1 -- 172.21.15.27:0/478789963 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8054153660 con 0x7f8054150a30 2023-10-06T01:41:49.164 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.156+0000 7f803bfff700 1 -- 172.21.15.27:0/478789963 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804c00fe20 con 0x7f8054150a30 2023-10-06T01:41:49.164 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.156+0000 7f805aa54700 1 -- 172.21.15.27:0/478789963 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8054151c30 con 0x7f8054150a30 2023-10-06T01:41:49.164 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.157+0000 7f805aa54700 1 -- 172.21.15.27:0/478789963 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8054064050 con 0x7f8054150a30 2023-10-06T01:41:49.165 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.157+0000 7f803bfff700 1 -- 172.21.15.27:0/478789963 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f804c015640 con 0x7f8054150a30 2023-10-06T01:41:49.165 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.157+0000 7f803bfff700 1 --2- 172.21.15.27:0/478789963 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8040066f30 0x7f80400693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:49.165 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.158+0000 7f803bfff700 1 -- 172.21.15.27:0/478789963 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f804c085070 con 0x7f8054150a30 2023-10-06T01:41:49.165 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.158+0000 7f8053fff700 1 --2- 172.21.15.27:0/478789963 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8040066f30 0x7f80400693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:49.165 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.158+0000 7f8053fff700 1 --2- 172.21.15.27:0/478789963 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8040066f30 0x7f80400693e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f804400c1e0 tx=0x7f8044015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:49.165 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.162+0000 7f803bfff700 1 -- 172.21.15.27:0/478789963 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f804c0520e0 con 0x7f8054150a30 2023-10-06T01:41:49.355 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.355+0000 7f805aa54700 1 -- 172.21.15.27:0/478789963 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f80541537f0 con 0x7f8054150a30 2023-10-06T01:41:49.355 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.355+0000 7f803bfff700 1 -- 172.21.15.27:0/478789963 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) v1 ==== 92+0+10 (secure 0 0 0) 0x7f804c055700 con 0x7f8054150a30 2023-10-06T01:41:49.355 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 8 2023-10-06T01:41:49.358 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.357+0000 7f805aa54700 1 -- 172.21.15.27:0/478789963 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8040066f30 msgr2=0x7f80400693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:49.358 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.357+0000 7f805aa54700 1 --2- 172.21.15.27:0/478789963 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8040066f30 0x7f80400693e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f804400c1e0 tx=0x7f8044015000 comp rx=0 tx=0).stop 2023-10-06T01:41:49.358 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.357+0000 7f805aa54700 1 -- 172.21.15.27:0/478789963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8054150a30 msgr2=0x7f8054150ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:49.358 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.357+0000 7f805aa54700 1 --2- 172.21.15.27:0/478789963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8054150a30 0x7f8054150ed0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f804c0094d0 tx=0x7f804c00b040 comp rx=0 tx=0).stop 2023-10-06T01:41:49.358 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.357+0000 7f805aa54700 1 -- 172.21.15.27:0/478789963 shutdown_connections 2023-10-06T01:41:49.359 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.357+0000 7f805aa54700 1 --2- 172.21.15.27:0/478789963 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8040066f30 0x7f80400693e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:49.359 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.357+0000 7f805aa54700 1 --2- 172.21.15.27:0/478789963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8054150a30 0x7f8054150ed0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:49.359 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.357+0000 7f805aa54700 1 --2- 172.21.15.27:0/478789963 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f803c003c20 0x7f8054141fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:49.359 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.357+0000 7f805aa54700 1 --2- 172.21.15.27:0/478789963 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f80540ceb80 0x7f80540cefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:49.359 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.357+0000 7f805aa54700 1 -- 172.21.15.27:0/478789963 >> 172.21.15.27:0/478789963 conn(0x7f8054067630 msgr2=0x7f805414f900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:49.359 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.357+0000 7f805aa54700 1 -- 172.21.15.27:0/478789963 shutdown_connections 2023-10-06T01:41:49.359 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.357+0000 7f805aa54700 1 -- 172.21.15.27:0/478789963 wait complete. 2023-10-06T01:41:49.369 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-06T01:41:49.492 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.491+0000 7f8652609700 1 Processor -- start 2023-10-06T01:41:49.492 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.491+0000 7f8652609700 1 -- start start 2023-10-06T01:41:49.500 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.492+0000 7f8652609700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f864c06b470 0x7f864c15d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:49.500 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.492+0000 7f8652609700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f864c06f270 0x7f864c0c79c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:49.500 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.492+0000 7f8652609700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f864c0ae340 con 0x7f864c0c7f00 2023-10-06T01:41:49.500 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.492+0000 7f8652609700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f864c15e290 con 0x7f864c06f270 2023-10-06T01:41:49.500 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.492+0000 7f8652609700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f864c0c8870 con 0x7f864c06b470 2023-10-06T01:41:49.500 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.492+0000 7f864b7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f864c06f270 0x7f864c0c79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:49.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.492+0000 7f8650ba6700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f864c0c7f00 0x7f864c0aea90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38474/0 (socket says 172.21.15.27:38474) 2023-10-06T01:41:49.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.492+0000 7f8650ba6700 1 -- 172.21.15.27:0/2803914278 learned_addr learned my addr 172.21.15.27:0/2803914278 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:49.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.492+0000 7f864b7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f864c06f270 0x7f864c0c79c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39496/0 (socket says 172.21.15.27:39496) 2023-10-06T01:41:49.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.493+0000 7f864bfff700 1 --2- 172.21.15.27:0/2803914278 >> v2:172.21.15.124:3300/0 conn(0x7f864c06b470 0x7f864c15d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:49.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.493+0000 7f864b7fe700 1 -- 172.21.15.27:0/2803914278 >> v2:172.21.15.124:3300/0 conn(0x7f864c06b470 msgr2=0x7f864c15d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:49.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.493+0000 7f864b7fe700 1 --2- 172.21.15.27:0/2803914278 >> v2:172.21.15.124:3300/0 conn(0x7f864c06b470 0x7f864c15d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:49.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.493+0000 7f864b7fe700 1 -- 172.21.15.27:0/2803914278 >> v1:172.21.15.27:6789/0 conn(0x7f864c0c7f00 legacy=0x7f864c0aea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:49.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.493+0000 7f864b7fe700 1 -- 172.21.15.27:0/2803914278 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f864c0c8340 con 0x7f864c06f270 2023-10-06T01:41:49.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.493+0000 7f864affd700 1 -- 172.21.15.27:0/2803914278 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3515696905 0 0) 0x7f864c0ae340 con 0x7f864c0c7f00 2023-10-06T01:41:49.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.493+0000 7f864bfff700 1 --2- 172.21.15.27:0/2803914278 >> v2:172.21.15.124:3300/0 conn(0x7f864c06b470 0x7f864c15d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:41:49.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.493+0000 7f864b7fe700 1 --2- 172.21.15.27:0/2803914278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f864c06f270 0x7f864c0c79c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f864c06e560 tx=0x7f86340091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=952170d9b86c9188 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:49.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.493+0000 7f864affd700 1 -- 172.21.15.27:0/2803914278 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863400b6b0 con 0x7f864c06f270 2023-10-06T01:41:49.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.493+0000 7f864affd700 1 -- 172.21.15.27:0/2803914278 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f863400b810 con 0x7f864c06f270 2023-10-06T01:41:49.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.493+0000 7f864affd700 1 -- 172.21.15.27:0/2803914278 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863400bbb0 con 0x7f864c06f270 2023-10-06T01:41:49.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.494+0000 7f8652609700 1 -- 172.21.15.27:0/2803914278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f864c06f270 msgr2=0x7f864c0c79c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:49.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.494+0000 7f8652609700 1 --2- 172.21.15.27:0/2803914278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f864c06f270 0x7f864c0c79c0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f864c06e560 tx=0x7f86340091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:49.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.494+0000 7f8652609700 1 -- 172.21.15.27:0/2803914278 shutdown_connections 2023-10-06T01:41:49.503 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.494+0000 7f8652609700 1 --2- 172.21.15.27:0/2803914278 >> v2:172.21.15.124:3300/0 conn(0x7f864c06b470 0x7f864c15d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:49.503 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.494+0000 7f8652609700 1 --2- 172.21.15.27:0/2803914278 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f864c06f270 0x7f864c0c79c0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:49.503 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.494+0000 7f8652609700 1 -- 172.21.15.27:0/2803914278 >> 172.21.15.27:0/2803914278 conn(0x7f864c06e760 msgr2=0x7f864c06eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:49.503 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.494+0000 7f8652609700 1 -- 172.21.15.27:0/2803914278 shutdown_connections 2023-10-06T01:41:49.504 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.494+0000 7f8652609700 1 -- 172.21.15.27:0/2803914278 wait complete. 2023-10-06T01:41:49.504 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.495+0000 7f8652609700 1 Processor -- start 2023-10-06T01:41:49.504 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.495+0000 7f8652609700 1 -- start start 2023-10-06T01:41:49.504 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.496+0000 7f8652609700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f864c06b470 0x7f864c06a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:49.504 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.496+0000 7f8652609700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f864c06f270 0x7f864c0b1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:49.505 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.496+0000 7f8652609700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f864c0c7f00 0x7f864c0b2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:49.505 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.496+0000 7f8652609700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864c0c6e00 con 0x7f864c06b470 2023-10-06T01:41:49.505 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.496+0000 7f8652609700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864c0b0470 con 0x7f864c0c7f00 2023-10-06T01:41:49.505 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.496+0000 7f8652609700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f864c158520 con 0x7f864c06f270 2023-10-06T01:41:49.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.496+0000 7f864bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f864c06b470 0x7f864c06a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:49.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.496+0000 7f864bfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f864c06b470 0x7f864c06a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39504/0 (socket says 172.21.15.27:39504) 2023-10-06T01:41:49.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.496+0000 7f864bfff700 1 -- 172.21.15.27:0/1936626604 learned_addr learned my addr 172.21.15.27:0/1936626604 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:49.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.496+0000 7f864b7fe700 1 --2- 172.21.15.27:0/1936626604 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f864c06f270 0x7f864c0b1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:49.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.496+0000 7f8650ba6700 1 --2- 172.21.15.27:0/1936626604 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f864c0c7f00 0x7f864c0b2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:49.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.496+0000 7f864bfff700 1 -- 172.21.15.27:0/1936626604 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f864c06f270 msgr2=0x7f864c0b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:49.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.496+0000 7f864bfff700 1 --2- 172.21.15.27:0/1936626604 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f864c06f270 0x7f864c0b1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:49.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.496+0000 7f864bfff700 1 -- 172.21.15.27:0/1936626604 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f864c0c7f00 msgr2=0x7f864c0b2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:49.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.496+0000 7f864bfff700 1 --2- 172.21.15.27:0/1936626604 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f864c0c7f00 0x7f864c0b2030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:49.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.496+0000 7f864bfff700 1 -- 172.21.15.27:0/1936626604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8634009420 con 0x7f864c06b470 2023-10-06T01:41:49.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.496+0000 7f864bfff700 1 --2- 172.21.15.27:0/1936626604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f864c06b470 0x7f864c06a6e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f863c00c830 tx=0x7f863c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:49.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.496+0000 7f8643fff700 1 -- 172.21.15.27:0/1936626604 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863c011600 con 0x7f864c06b470 2023-10-06T01:41:49.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.496+0000 7f8643fff700 1 -- 172.21.15.27:0/1936626604 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f863c011760 con 0x7f864c06b470 2023-10-06T01:41:49.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.497+0000 7f8652609700 1 -- 172.21.15.27:0/1936626604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f864c0c61f0 con 0x7f864c06b470 2023-10-06T01:41:49.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.497+0000 7f8643fff700 1 -- 172.21.15.27:0/1936626604 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863c011b20 con 0x7f864c06b470 2023-10-06T01:41:49.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.497+0000 7f8652609700 1 -- 172.21.15.27:0/1936626604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f864c0c67b0 con 0x7f864c06b470 2023-10-06T01:41:49.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.497+0000 7f8652609700 1 -- 172.21.15.27:0/1936626604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f864c0cd6f0 con 0x7f864c06b470 2023-10-06T01:41:49.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.498+0000 7f8643fff700 1 -- 172.21.15.27:0/1936626604 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f863c00a3e0 con 0x7f864c06b470 2023-10-06T01:41:49.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.498+0000 7f8643fff700 1 --2- 172.21.15.27:0/1936626604 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f862c066f30 0x7f862c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:49.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.498+0000 7f8643fff700 1 -- 172.21.15.27:0/1936626604 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f863c084b50 con 0x7f864c06b470 2023-10-06T01:41:49.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.498+0000 7f864b7fe700 1 --2- 172.21.15.27:0/1936626604 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f862c066f30 0x7f862c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:49.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.499+0000 7f864b7fe700 1 --2- 172.21.15.27:0/1936626604 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f862c066f30 0x7f862c0693e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f864c0c7430 tx=0x7f8634017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:49.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.503+0000 7f8643fff700 1 -- 172.21.15.27:0/1936626604 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f863c051c60 con 0x7f864c06b470 2023-10-06T01:41:49.694 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.694+0000 7f8652609700 1 -- 172.21.15.27:0/1936626604 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f864c064050 con 0x7f864c06b470 2023-10-06T01:41:49.694 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.694+0000 7f8643fff700 1 -- 172.21.15.27:0/1936626604 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f863c0150a0 con 0x7f864c06b470 2023-10-06T01:41:49.695 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2023-10-06T01:41:49.696 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.696+0000 7f8652609700 1 -- 172.21.15.27:0/1936626604 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f862c066f30 msgr2=0x7f862c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:49.697 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.696+0000 7f8652609700 1 --2- 172.21.15.27:0/1936626604 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f862c066f30 0x7f862c0693e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f864c0c7430 tx=0x7f8634017000 comp rx=0 tx=0).stop 2023-10-06T01:41:49.697 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.696+0000 7f8652609700 1 -- 172.21.15.27:0/1936626604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f864c06b470 msgr2=0x7f864c06a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:49.697 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.696+0000 7f8652609700 1 --2- 172.21.15.27:0/1936626604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f864c06b470 0x7f864c06a6e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f863c00c830 tx=0x7f863c008040 comp rx=0 tx=0).stop 2023-10-06T01:41:49.697 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.696+0000 7f8652609700 1 -- 172.21.15.27:0/1936626604 shutdown_connections 2023-10-06T01:41:49.697 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.696+0000 7f8652609700 1 --2- 172.21.15.27:0/1936626604 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f862c066f30 0x7f862c0693e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:49.697 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.696+0000 7f8652609700 1 --2- 172.21.15.27:0/1936626604 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f864c0c7f00 0x7f864c0b2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:49.697 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.696+0000 7f8652609700 1 --2- 172.21.15.27:0/1936626604 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f864c06b470 0x7f864c06a6e0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:49.697 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.696+0000 7f8652609700 1 --2- 172.21.15.27:0/1936626604 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f864c06f270 0x7f864c0b1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:49.697 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.696+0000 7f8652609700 1 -- 172.21.15.27:0/1936626604 >> 172.21.15.27:0/1936626604 conn(0x7f864c06e760 msgr2=0x7f864c0c2240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:49.697 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.696+0000 7f8652609700 1 -- 172.21.15.27:0/1936626604 shutdown_connections 2023-10-06T01:41:49.697 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.696+0000 7f8652609700 1 -- 172.21.15.27:0/1936626604 wait complete. 2023-10-06T01:41:49.708 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-06T01:41:49.830 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.830+0000 7efe6fb82700 1 Processor -- start 2023-10-06T01:41:49.830 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.830+0000 7efe6fb82700 1 -- start start 2023-10-06T01:41:49.833 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.831+0000 7efe6fb82700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7efe68159500 0x7efe681598d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:49.833 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.831+0000 7efe6fb82700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe680b1110 con 0x7efe6815dab0 2023-10-06T01:41:49.833 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.831+0000 7efe6fb82700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe680b13b0 con 0x7efe680b3fd0 2023-10-06T01:41:49.833 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.831+0000 7efe6fb82700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe680b1690 con 0x7efe68159500 2023-10-06T01:41:49.833 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.831+0000 7efe6e11f700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7efe6815dab0 0x7efe680cf6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38480/0 (socket says 172.21.15.27:38480) 2023-10-06T01:41:49.833 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.831+0000 7efe6e11f700 1 -- 172.21.15.27:0/3756473040 learned_addr learned my addr 172.21.15.27:0/3756473040 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:49.833 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.831+0000 7efe6d91e700 1 --2- 172.21.15.27:0/3756473040 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7efe68159500 0x7efe681598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:49.833 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.831+0000 7efe6c91c700 1 -- 172.21.15.27:0/3756473040 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3479951034 0 0) 0x7efe680b1110 con 0x7efe6815dab0 2023-10-06T01:41:49.834 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.831+0000 7efe6c91c700 1 -- 172.21.15.27:0/3756473040 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe50003570 con 0x7efe6815dab0 2023-10-06T01:41:49.834 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.831+0000 7efe6c91c700 1 -- 172.21.15.27:0/3756473040 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2782374750 0 0) 0x7efe680b13b0 con 0x7efe680b3fd0 2023-10-06T01:41:49.834 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.831+0000 7efe6c91c700 1 -- 172.21.15.27:0/3756473040 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe680b1110 con 0x7efe680b3fd0 2023-10-06T01:41:49.834 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.831+0000 7efe6d91e700 1 -- 172.21.15.27:0/3756473040 >> v1:172.21.15.124:6789/0 conn(0x7efe680b3fd0 legacy=0x7efe6815d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:49.834 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.831+0000 7efe6d91e700 1 -- 172.21.15.27:0/3756473040 >> v1:172.21.15.27:6789/0 conn(0x7efe6815dab0 legacy=0x7efe680cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:49.834 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.831+0000 7efe6d91e700 1 -- 172.21.15.27:0/3756473040 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe680b1970 con 0x7efe68159500 2023-10-06T01:41:49.834 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.832+0000 7efe6d91e700 1 --2- 172.21.15.27:0/3756473040 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7efe68159500 0x7efe681598d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7efe6815e3c0 tx=0x7efe580091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f720c119b24bef25 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:49.834 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.832+0000 7efe6c91c700 1 -- 172.21.15.27:0/3756473040 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe58011630 con 0x7efe68159500 2023-10-06T01:41:49.834 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.832+0000 7efe6c91c700 1 -- 172.21.15.27:0/3756473040 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe58011790 con 0x7efe68159500 2023-10-06T01:41:49.834 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.832+0000 7efe6c91c700 1 -- 172.21.15.27:0/3756473040 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe58011b50 con 0x7efe68159500 2023-10-06T01:41:49.835 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.832+0000 7efe6fb82700 1 -- 172.21.15.27:0/3756473040 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7efe68159500 msgr2=0x7efe681598d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:49.835 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.832+0000 7efe6fb82700 1 --2- 172.21.15.27:0/3756473040 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7efe68159500 0x7efe681598d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7efe6815e3c0 tx=0x7efe580091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:49.835 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.832+0000 7efe6fb82700 1 -- 172.21.15.27:0/3756473040 shutdown_connections 2023-10-06T01:41:49.835 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.832+0000 7efe6fb82700 1 --2- 172.21.15.27:0/3756473040 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7efe68159500 0x7efe681598d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:49.835 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.832+0000 7efe6fb82700 1 -- 172.21.15.27:0/3756473040 >> 172.21.15.27:0/3756473040 conn(0x7efe68067630 msgr2=0x7efe68069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:49.835 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.832+0000 7efe6fb82700 1 -- 172.21.15.27:0/3756473040 shutdown_connections 2023-10-06T01:41:49.835 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.832+0000 7efe6fb82700 1 -- 172.21.15.27:0/3756473040 wait complete. 2023-10-06T01:41:49.837 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.833+0000 7efe6fb82700 1 Processor -- start 2023-10-06T01:41:49.837 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.834+0000 7efe6fb82700 1 -- start start 2023-10-06T01:41:49.838 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.834+0000 7efe6fb82700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7efe680b3fd0 0x7efe68167bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:49.838 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.834+0000 7efe6fb82700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7efe68159500 0x7efe681548b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:49.838 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.834+0000 7efe6fb82700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe6815dab0 0x7efe6816b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:49.838 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.834+0000 7efe6fb82700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe68169050 con 0x7efe6815dab0 2023-10-06T01:41:49.838 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.834+0000 7efe6fb82700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe68168440 con 0x7efe680b3fd0 2023-10-06T01:41:49.838 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.834+0000 7efe6fb82700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe68168720 con 0x7efe68159500 2023-10-06T01:41:49.838 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.834+0000 7efe6e11f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe6815dab0 0x7efe6816b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:49.838 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.834+0000 7efe6e11f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe6815dab0 0x7efe6816b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39508/0 (socket says 172.21.15.27:39508) 2023-10-06T01:41:49.838 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.834+0000 7efe6e11f700 1 -- 172.21.15.27:0/2421677435 learned_addr learned my addr 172.21.15.27:0/2421677435 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:49.838 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.835+0000 7efe6d91e700 1 --2- 172.21.15.27:0/2421677435 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7efe680b3fd0 0x7efe68167bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:49.838 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.835+0000 7efe6d11d700 1 --2- 172.21.15.27:0/2421677435 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7efe68159500 0x7efe681548b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:49.839 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.835+0000 7efe6e11f700 1 -- 172.21.15.27:0/2421677435 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7efe68159500 msgr2=0x7efe681548b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:49.839 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.835+0000 7efe6e11f700 1 --2- 172.21.15.27:0/2421677435 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7efe68159500 0x7efe681548b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:49.839 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.835+0000 7efe6e11f700 1 -- 172.21.15.27:0/2421677435 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7efe680b3fd0 msgr2=0x7efe68167bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:49.839 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.835+0000 7efe6e11f700 1 --2- 172.21.15.27:0/2421677435 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7efe680b3fd0 0x7efe68167bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:49.839 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.835+0000 7efe6e11f700 1 -- 172.21.15.27:0/2421677435 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe58009420 con 0x7efe6815dab0 2023-10-06T01:41:49.839 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.835+0000 7efe6e11f700 1 --2- 172.21.15.27:0/2421677435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe6815dab0 0x7efe6816b0e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7efe60001460 tx=0x7efe60001a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:49.839 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.835+0000 7efe65ffb700 1 -- 172.21.15.27:0/2421677435 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6000a910 con 0x7efe6815dab0 2023-10-06T01:41:49.839 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.835+0000 7efe65ffb700 1 -- 172.21.15.27:0/2421677435 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe6000aa70 con 0x7efe6815dab0 2023-10-06T01:41:49.839 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.835+0000 7efe6fb82700 1 -- 172.21.15.27:0/2421677435 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe68168a00 con 0x7efe6815dab0 2023-10-06T01:41:49.839 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.835+0000 7efe65ffb700 1 -- 172.21.15.27:0/2421677435 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6000ae30 con 0x7efe6815dab0 2023-10-06T01:41:49.839 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.835+0000 7efe6fb82700 1 -- 172.21.15.27:0/2421677435 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe68168130 con 0x7efe6815dab0 2023-10-06T01:41:49.840 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.836+0000 7efe6fb82700 1 -- 172.21.15.27:0/2421677435 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe68064050 con 0x7efe6815dab0 2023-10-06T01:41:49.840 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.837+0000 7efe65ffb700 1 -- 172.21.15.27:0/2421677435 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7efe60007300 con 0x7efe6815dab0 2023-10-06T01:41:49.840 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.837+0000 7efe65ffb700 1 --2- 172.21.15.27:0/2421677435 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7efe54066f30 0x7efe540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:49.840 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.837+0000 7efe65ffb700 1 -- 172.21.15.27:0/2421677435 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7efe60084a60 con 0x7efe6815dab0 2023-10-06T01:41:49.840 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.837+0000 7efe6d91e700 1 --2- 172.21.15.27:0/2421677435 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7efe54066f30 0x7efe540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:49.840 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.838+0000 7efe6d91e700 1 --2- 172.21.15.27:0/2421677435 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7efe54066f30 0x7efe540693e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7efe58009860 tx=0x7efe58016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:49.841 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:49.841+0000 7efe65ffb700 1 -- 172.21.15.27:0/2421677435 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efe60051bf0 con 0x7efe6815dab0 2023-10-06T01:41:50.033 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.033+0000 7efe6fb82700 1 -- 172.21.15.27:0/2421677435 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efe680d0420 con 0x7efe6815dab0 2023-10-06T01:41:50.033 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.033+0000 7efe65ffb700 1 -- 172.21.15.27:0/2421677435 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7efe60055210 con 0x7efe6815dab0 2023-10-06T01:41:50.033 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2023-10-06T01:41:50.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.035+0000 7efe6fb82700 1 -- 172.21.15.27:0/2421677435 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7efe54066f30 msgr2=0x7efe540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:50.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.035+0000 7efe6fb82700 1 --2- 172.21.15.27:0/2421677435 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7efe54066f30 0x7efe540693e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7efe58009860 tx=0x7efe58016000 comp rx=0 tx=0).stop 2023-10-06T01:41:50.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.035+0000 7efe6fb82700 1 -- 172.21.15.27:0/2421677435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe6815dab0 msgr2=0x7efe6816b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:50.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.035+0000 7efe6fb82700 1 --2- 172.21.15.27:0/2421677435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe6815dab0 0x7efe6816b0e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7efe60001460 tx=0x7efe60001a20 comp rx=0 tx=0).stop 2023-10-06T01:41:50.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.035+0000 7efe6fb82700 1 -- 172.21.15.27:0/2421677435 shutdown_connections 2023-10-06T01:41:50.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.035+0000 7efe6fb82700 1 --2- 172.21.15.27:0/2421677435 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7efe54066f30 0x7efe540693e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:50.037 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.035+0000 7efe6fb82700 1 --2- 172.21.15.27:0/2421677435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efe6815dab0 0x7efe6816b0e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:50.037 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.035+0000 7efe6fb82700 1 --2- 172.21.15.27:0/2421677435 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7efe680b3fd0 0x7efe68167bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:50.037 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.035+0000 7efe6fb82700 1 --2- 172.21.15.27:0/2421677435 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7efe68159500 0x7efe681548b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:50.037 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.035+0000 7efe6fb82700 1 -- 172.21.15.27:0/2421677435 >> 172.21.15.27:0/2421677435 conn(0x7efe68067630 msgr2=0x7efe68069a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:50.037 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.035+0000 7efe6fb82700 1 -- 172.21.15.27:0/2421677435 shutdown_connections 2023-10-06T01:41:50.037 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.035+0000 7efe6fb82700 1 -- 172.21.15.27:0/2421677435 wait complete. 2023-10-06T01:41:50.048 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-06T01:41:50.169 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.169+0000 7f03e50b6700 1 Processor -- start 2023-10-06T01:41:50.169 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.169+0000 7f03e50b6700 1 -- start start 2023-10-06T01:41:50.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.169+0000 7f03e50b6700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f03e01598d0 0x7f03e015d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:50.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.169+0000 7f03e50b6700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03e015dd20 con 0x7f03e0159500 2023-10-06T01:41:50.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.169+0000 7f03e50b6700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03e00b1110 con 0x7f03e015d8e0 2023-10-06T01:41:50.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.169+0000 7f03e50b6700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03e00b13b0 con 0x7f03e01598d0 2023-10-06T01:41:50.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.170+0000 7f03ded9d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f03e0159500 0x7f03e00b3fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38496/0 (socket says 172.21.15.27:38496) 2023-10-06T01:41:50.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.170+0000 7f03ded9d700 1 -- 172.21.15.27:0/2323052421 learned_addr learned my addr 172.21.15.27:0/2323052421 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:50.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.170+0000 7f03de59c700 1 --2- 172.21.15.27:0/2323052421 >> v2:172.21.15.124:3300/0 conn(0x7f03e01598d0 0x7f03e015d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:50.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.170+0000 7f03ddd9b700 1 -- 172.21.15.27:0/2323052421 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3425583263 0 0) 0x7f03e015dd20 con 0x7f03e0159500 2023-10-06T01:41:50.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.170+0000 7f03ddd9b700 1 -- 172.21.15.27:0/2323052421 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03c8003570 con 0x7f03e0159500 2023-10-06T01:41:50.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.170+0000 7f03de59c700 1 -- 172.21.15.27:0/2323052421 >> v1:172.21.15.124:6789/0 conn(0x7f03e015d8e0 legacy=0x7f03e00cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:50.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.170+0000 7f03ddd9b700 1 -- 172.21.15.27:0/2323052421 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2136899845 0 0) 0x7f03e00b1110 con 0x7f03e015d8e0 2023-10-06T01:41:50.173 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.170+0000 7f03de59c700 1 -- 172.21.15.27:0/2323052421 >> v1:172.21.15.27:6789/0 conn(0x7f03e0159500 legacy=0x7f03e00b3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:50.173 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.170+0000 7f03de59c700 1 -- 172.21.15.27:0/2323052421 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e00b1690 con 0x7f03e01598d0 2023-10-06T01:41:50.173 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.170+0000 7f03de59c700 1 --2- 172.21.15.27:0/2323052421 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f03e01598d0 0x7f03e015d3a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f03d40098e0 tx=0x7f03d40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1434e065e61d65dd server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:50.173 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.171+0000 7f03ddd9b700 1 -- 172.21.15.27:0/2323052421 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d400b640 con 0x7f03e01598d0 2023-10-06T01:41:50.173 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.171+0000 7f03ddd9b700 1 -- 172.21.15.27:0/2323052421 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03d400b7a0 con 0x7f03e01598d0 2023-10-06T01:41:50.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.171+0000 7f03ddd9b700 1 -- 172.21.15.27:0/2323052421 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d400bb40 con 0x7f03e01598d0 2023-10-06T01:41:50.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.171+0000 7f03e50b6700 1 -- 172.21.15.27:0/2323052421 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f03e01598d0 msgr2=0x7f03e015d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:50.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.171+0000 7f03e50b6700 1 --2- 172.21.15.27:0/2323052421 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f03e01598d0 0x7f03e015d3a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f03d40098e0 tx=0x7f03d40091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:50.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.171+0000 7f03e50b6700 1 -- 172.21.15.27:0/2323052421 shutdown_connections 2023-10-06T01:41:50.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.171+0000 7f03e50b6700 1 --2- 172.21.15.27:0/2323052421 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f03e01598d0 0x7f03e015d3a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:50.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.171+0000 7f03e50b6700 1 -- 172.21.15.27:0/2323052421 >> 172.21.15.27:0/2323052421 conn(0x7f03e0067630 msgr2=0x7f03e0069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:50.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.171+0000 7f03e50b6700 1 -- 172.21.15.27:0/2323052421 shutdown_connections 2023-10-06T01:41:50.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.171+0000 7f03e50b6700 1 -- 172.21.15.27:0/2323052421 wait complete. 2023-10-06T01:41:50.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.172+0000 7f03e50b6700 1 Processor -- start 2023-10-06T01:41:50.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.172+0000 7f03e50b6700 1 -- start start 2023-10-06T01:41:50.179 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.173+0000 7f03e50b6700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f03e0159500 0x7f03e0139780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:50.179 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.173+0000 7f03e50b6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e01598d0 0x7f03e014cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:50.179 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.173+0000 7f03e50b6700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f03e015d8e0 0x7f03e00689d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:50.179 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.173+0000 7f03e50b6700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e014d510 con 0x7f03e01598d0 2023-10-06T01:41:50.179 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.173+0000 7f03e50b6700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e014d7f0 con 0x7f03e0159500 2023-10-06T01:41:50.180 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.173+0000 7f03e50b6700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03e014dad0 con 0x7f03e015d8e0 2023-10-06T01:41:50.180 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.173+0000 7f03de59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e01598d0 0x7f03e014cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:50.180 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.173+0000 7f03de59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e01598d0 0x7f03e014cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39522/0 (socket says 172.21.15.27:39522) 2023-10-06T01:41:50.180 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.173+0000 7f03de59c700 1 -- 172.21.15.27:0/2812795888 learned_addr learned my addr 172.21.15.27:0/2812795888 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:50.181 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.173+0000 7f03ded9d700 1 --2- 172.21.15.27:0/2812795888 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f03e0159500 0x7f03e0139780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:50.181 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.173+0000 7f03df59e700 1 --2- 172.21.15.27:0/2812795888 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f03e015d8e0 0x7f03e00689d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:50.181 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.173+0000 7f03de59c700 1 -- 172.21.15.27:0/2812795888 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f03e015d8e0 msgr2=0x7f03e00689d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:50.181 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.173+0000 7f03de59c700 1 --2- 172.21.15.27:0/2812795888 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f03e015d8e0 0x7f03e00689d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:50.181 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.174+0000 7f03de59c700 1 -- 172.21.15.27:0/2812795888 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f03e0159500 msgr2=0x7f03e0139780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:50.182 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.174+0000 7f03de59c700 1 --2- 172.21.15.27:0/2812795888 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f03e0159500 0x7f03e0139780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:50.182 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.174+0000 7f03de59c700 1 -- 172.21.15.27:0/2812795888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d4009420 con 0x7f03e01598d0 2023-10-06T01:41:50.182 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.174+0000 7f03de59c700 1 --2- 172.21.15.27:0/2812795888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e01598d0 0x7f03e014cae0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f03e015deb0 tx=0x7f03d4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:50.182 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.174+0000 7f03c6ffd700 1 -- 172.21.15.27:0/2812795888 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d401f070 con 0x7f03e01598d0 2023-10-06T01:41:50.182 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.174+0000 7f03c6ffd700 1 -- 172.21.15.27:0/2812795888 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03d4000c90 con 0x7f03e01598d0 2023-10-06T01:41:50.183 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.174+0000 7f03e50b6700 1 -- 172.21.15.27:0/2812795888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03e014ddb0 con 0x7f03e01598d0 2023-10-06T01:41:50.183 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.174+0000 7f03c6ffd700 1 -- 172.21.15.27:0/2812795888 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03d400bb40 con 0x7f03e01598d0 2023-10-06T01:41:50.183 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.174+0000 7f03e50b6700 1 -- 172.21.15.27:0/2812795888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03e0139cc0 con 0x7f03e01598d0 2023-10-06T01:41:50.183 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.175+0000 7f03e50b6700 1 -- 172.21.15.27:0/2812795888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03e0088d80 con 0x7f03e01598d0 2023-10-06T01:41:50.184 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.175+0000 7f03c6ffd700 1 -- 172.21.15.27:0/2812795888 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f03d4023070 con 0x7f03e01598d0 2023-10-06T01:41:50.184 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.176+0000 7f03c6ffd700 1 --2- 172.21.15.27:0/2812795888 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f03cc066f30 0x7f03cc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:50.184 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.176+0000 7f03c6ffd700 1 -- 172.21.15.27:0/2812795888 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f03d4085b60 con 0x7f03e01598d0 2023-10-06T01:41:50.184 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.176+0000 7f03ded9d700 1 --2- 172.21.15.27:0/2812795888 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f03cc066f30 0x7f03cc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:50.184 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.176+0000 7f03ded9d700 1 --2- 172.21.15.27:0/2812795888 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f03cc066f30 0x7f03cc0693e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f03d00058d0 tx=0x7f03d0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:50.185 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.180+0000 7f03c6ffd700 1 -- 172.21.15.27:0/2812795888 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f03d4052c70 con 0x7f03e01598d0 2023-10-06T01:41:50.373 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.373+0000 7f03e50b6700 1 -- 172.21.15.27:0/2812795888 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f03e0064050 con 0x7f03e01598d0 2023-10-06T01:41:50.373 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.373+0000 7f03c6ffd700 1 -- 172.21.15.27:0/2812795888 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f03d40150a0 con 0x7f03e01598d0 2023-10-06T01:41:50.373 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2023-10-06T01:41:50.376 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.375+0000 7f03e50b6700 1 -- 172.21.15.27:0/2812795888 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f03cc066f30 msgr2=0x7f03cc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:50.376 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.375+0000 7f03e50b6700 1 --2- 172.21.15.27:0/2812795888 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f03cc066f30 0x7f03cc0693e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f03d00058d0 tx=0x7f03d0009040 comp rx=0 tx=0).stop 2023-10-06T01:41:50.376 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.375+0000 7f03e50b6700 1 -- 172.21.15.27:0/2812795888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e01598d0 msgr2=0x7f03e014cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:50.376 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.375+0000 7f03e50b6700 1 --2- 172.21.15.27:0/2812795888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e01598d0 0x7f03e014cae0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f03e015deb0 tx=0x7f03d4017000 comp rx=0 tx=0).stop 2023-10-06T01:41:50.376 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.375+0000 7f03e50b6700 1 -- 172.21.15.27:0/2812795888 shutdown_connections 2023-10-06T01:41:50.376 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.375+0000 7f03e50b6700 1 --2- 172.21.15.27:0/2812795888 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f03cc066f30 0x7f03cc0693e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:50.376 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.375+0000 7f03e50b6700 1 --2- 172.21.15.27:0/2812795888 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f03e015d8e0 0x7f03e00689d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:50.376 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.375+0000 7f03e50b6700 1 --2- 172.21.15.27:0/2812795888 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f03e0159500 0x7f03e0139780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:50.376 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.375+0000 7f03e50b6700 1 --2- 172.21.15.27:0/2812795888 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f03e01598d0 0x7f03e014cae0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:50.376 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.375+0000 7f03e50b6700 1 -- 172.21.15.27:0/2812795888 >> 172.21.15.27:0/2812795888 conn(0x7f03e0067630 msgr2=0x7f03e014c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:50.376 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.375+0000 7f03e50b6700 1 -- 172.21.15.27:0/2812795888 shutdown_connections 2023-10-06T01:41:50.377 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.375+0000 7f03e50b6700 1 -- 172.21.15.27:0/2812795888 wait complete. 2023-10-06T01:41:50.388 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T01:41:50.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.507+0000 7f54ed0d0700 1 Processor -- start 2023-10-06T01:41:50.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.507+0000 7f54ed0d0700 1 -- start start 2023-10-06T01:41:50.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.508+0000 7f54ed0d0700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f54e815d930 0x7f54e80d0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:50.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.508+0000 7f54ed0d0700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f54e81591b0 0x7f54e8159630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:50.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.508+0000 7f54ed0d0700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54e80b2ad0 con 0x7f54e815d560 2023-10-06T01:41:50.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.508+0000 7f54ed0d0700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54e80b4160 con 0x7f54e815d930 2023-10-06T01:41:50.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.508+0000 7f54ed0d0700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54e8159b70 con 0x7f54e81591b0 2023-10-06T01:41:50.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.508+0000 7f54e759e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f54e81591b0 0x7f54e8159630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:50.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.508+0000 7f54e659c700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f54e815d930 0x7f54e80d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:50.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.508+0000 7f54e759e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f54e81591b0 0x7f54e8159630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.27:53220/0 (socket says 172.21.15.27:53220) 2023-10-06T01:41:50.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.508+0000 7f54e759e700 1 -- 172.21.15.27:0/2795760471 learned_addr learned my addr 172.21.15.27:0/2795760471 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:50.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.508+0000 7f54e5d9b700 1 -- 172.21.15.27:0/2795760471 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3656829232 0 0) 0x7f54e80b2ad0 con 0x7f54e815d560 2023-10-06T01:41:50.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.508+0000 7f54e5d9b700 1 -- 172.21.15.27:0/2795760471 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54d4003570 con 0x7f54e815d560 2023-10-06T01:41:50.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.508+0000 7f54e659c700 1 -- 172.21.15.27:0/2795760471 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f54e81591b0 msgr2=0x7f54e8159630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:50.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.508+0000 7f54e659c700 1 --2- 172.21.15.27:0/2795760471 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f54e81591b0 0x7f54e8159630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:50.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.508+0000 7f54e659c700 1 -- 172.21.15.27:0/2795760471 >> v1:172.21.15.124:6789/0 conn(0x7f54e815d560 legacy=0x7f54e80b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:50.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.508+0000 7f54e659c700 1 -- 172.21.15.27:0/2795760471 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e80b2230 con 0x7f54e815d930 2023-10-06T01:41:50.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.508+0000 7f54e759e700 1 --2- 172.21.15.27:0/2795760471 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f54e81591b0 0x7f54e8159630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:41:50.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.509+0000 7f54e659c700 1 --2- 172.21.15.27:0/2795760471 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54e815d930 0x7f54e80d0030 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f54d00098e0 tx=0x7f54d00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45585c6c42d748f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:50.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.509+0000 7f54e5d9b700 1 -- 172.21.15.27:0/2795760471 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d0011630 con 0x7f54e815d930 2023-10-06T01:41:50.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.509+0000 7f54e5d9b700 1 -- 172.21.15.27:0/2795760471 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54d0011790 con 0x7f54e815d930 2023-10-06T01:41:50.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.509+0000 7f54e5d9b700 1 -- 172.21.15.27:0/2795760471 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d0011b30 con 0x7f54e815d930 2023-10-06T01:41:50.513 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.509+0000 7f54ed0d0700 1 -- 172.21.15.27:0/2795760471 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54e815d930 msgr2=0x7f54e80d0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:50.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.509+0000 7f54ed0d0700 1 --2- 172.21.15.27:0/2795760471 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54e815d930 0x7f54e80d0030 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f54d00098e0 tx=0x7f54d00091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:50.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.509+0000 7f54ed0d0700 1 -- 172.21.15.27:0/2795760471 shutdown_connections 2023-10-06T01:41:50.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.509+0000 7f54ed0d0700 1 --2- 172.21.15.27:0/2795760471 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f54e81591b0 0x7f54e8159630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:50.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.509+0000 7f54ed0d0700 1 --2- 172.21.15.27:0/2795760471 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54e815d930 0x7f54e80d0030 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:50.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.509+0000 7f54ed0d0700 1 -- 172.21.15.27:0/2795760471 >> 172.21.15.27:0/2795760471 conn(0x7f54e80d1180 msgr2=0x7f54e8069640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:50.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.509+0000 7f54ed0d0700 1 -- 172.21.15.27:0/2795760471 shutdown_connections 2023-10-06T01:41:50.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.509+0000 7f54ed0d0700 1 -- 172.21.15.27:0/2795760471 wait complete. 2023-10-06T01:41:50.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.511+0000 7f54ed0d0700 1 Processor -- start 2023-10-06T01:41:50.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.511+0000 7f54ed0d0700 1 -- start start 2023-10-06T01:41:50.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.511+0000 7f54ed0d0700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54e81591b0 0x7f54e80aab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:50.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.511+0000 7f54ed0d0700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f54e815d560 0x7f54e816a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:50.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.511+0000 7f54ed0d0700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54e815d930 0x7f54e8155950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:50.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.511+0000 7f54ed0d0700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e816a8e0 con 0x7f54e815d930 2023-10-06T01:41:50.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.511+0000 7f54ed0d0700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e80abad0 con 0x7f54e81591b0 2023-10-06T01:41:50.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.511+0000 7f54ed0d0700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54e8155e90 con 0x7f54e815d560 2023-10-06T01:41:50.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.511+0000 7f54e759e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54e815d930 0x7f54e8155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:50.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.511+0000 7f54e759e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54e815d930 0x7f54e8155950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39538/0 (socket says 172.21.15.27:39538) 2023-10-06T01:41:50.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.511+0000 7f54e759e700 1 -- 172.21.15.27:0/340235093 learned_addr learned my addr 172.21.15.27:0/340235093 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:50.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.512+0000 7f54e659c700 1 --2- 172.21.15.27:0/340235093 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f54e815d560 0x7f54e816a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:50.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.512+0000 7f54e6d9d700 1 --2- 172.21.15.27:0/340235093 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54e81591b0 0x7f54e80aab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:50.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.512+0000 7f54e759e700 1 -- 172.21.15.27:0/340235093 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f54e815d560 msgr2=0x7f54e816a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:50.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.512+0000 7f54e759e700 1 --2- 172.21.15.27:0/340235093 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f54e815d560 0x7f54e816a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:50.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.512+0000 7f54e759e700 1 -- 172.21.15.27:0/340235093 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54e81591b0 msgr2=0x7f54e80aab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:50.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.512+0000 7f54e759e700 1 --2- 172.21.15.27:0/340235093 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54e81591b0 0x7f54e80aab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:50.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.512+0000 7f54e759e700 1 -- 172.21.15.27:0/340235093 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e00094f0 con 0x7f54e815d930 2023-10-06T01:41:50.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.512+0000 7f54e759e700 1 --2- 172.21.15.27:0/340235093 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54e815d930 0x7f54e8155950 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f54e000ec50 tx=0x7f54e0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:50.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.512+0000 7f54deffd700 1 -- 172.21.15.27:0/340235093 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e000c9d0 con 0x7f54e815d930 2023-10-06T01:41:50.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.512+0000 7f54deffd700 1 -- 172.21.15.27:0/340235093 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54e000cb30 con 0x7f54e815d930 2023-10-06T01:41:50.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.512+0000 7f54ed0d0700 1 -- 172.21.15.27:0/340235093 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54d0009420 con 0x7f54e815d930 2023-10-06T01:41:50.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.512+0000 7f54deffd700 1 -- 172.21.15.27:0/340235093 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e0013470 con 0x7f54e815d930 2023-10-06T01:41:50.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.512+0000 7f54ed0d0700 1 -- 172.21.15.27:0/340235093 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54e80ab510 con 0x7f54e815d930 2023-10-06T01:41:50.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.513+0000 7f54ed0d0700 1 -- 172.21.15.27:0/340235093 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54e80aeeb0 con 0x7f54e815d930 2023-10-06T01:41:50.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.514+0000 7f54deffd700 1 -- 172.21.15.27:0/340235093 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f54e00138a0 con 0x7f54e815d930 2023-10-06T01:41:50.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.514+0000 7f54deffd700 1 --2- 172.21.15.27:0/340235093 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f54c8066f30 0x7f54c80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:50.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.514+0000 7f54deffd700 1 -- 172.21.15.27:0/340235093 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f54e0084a00 con 0x7f54e815d930 2023-10-06T01:41:50.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.514+0000 7f54e6d9d700 1 --2- 172.21.15.27:0/340235093 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f54c8066f30 0x7f54c80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:50.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.515+0000 7f54e6d9d700 1 --2- 172.21.15.27:0/340235093 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f54c8066f30 0x7f54c80693e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f54e815c8c0 tx=0x7f54d8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:50.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.518+0000 7f54deffd700 1 -- 172.21.15.27:0/340235093 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f54e0051b10 con 0x7f54e815d930 2023-10-06T01:41:50.757 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.756+0000 7f54ed0d0700 1 -- 172.21.15.27:0/340235093 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54e8064050 con 0x7f54e815d930 2023-10-06T01:41:50.757 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.757+0000 7f54deffd700 1 -- 172.21.15.27:0/340235093 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4594 (secure 0 0 0) 0x7f54e0020090 con 0x7f54e815d930 2023-10-06T01:41:50.758 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:50.758 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4261,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6837/2699906167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6836","nonce":2699906167},{"type":"v1","addr":"172.21.15.124:6837","nonce":2699906167}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/313372271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":313372271},{"type":"v1","addr":"172.21.15.27:6834","nonce":313372271}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4287,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/4257743842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4257743842},{"type":"v1","addr":"172.21.15.27:6835","nonce":4257743842}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T01:41:12.735970+0000","modified":"2023-10-06T01:41:16.358924+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.124:6835/3803994994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":3803994994},{"type":"v1","addr":"172.21.15.124:6835","nonce":3803994994}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-06T01:41:50.759 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.758+0000 7f54ed0d0700 1 -- 172.21.15.27:0/340235093 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f54c8066f30 msgr2=0x7f54c80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:50.759 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.758+0000 7f54ed0d0700 1 --2- 172.21.15.27:0/340235093 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f54c8066f30 0x7f54c80693e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f54e815c8c0 tx=0x7f54d8009040 comp rx=0 tx=0).stop 2023-10-06T01:41:50.759 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.758+0000 7f54ed0d0700 1 -- 172.21.15.27:0/340235093 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54e815d930 msgr2=0x7f54e8155950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:50.759 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.758+0000 7f54ed0d0700 1 --2- 172.21.15.27:0/340235093 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54e815d930 0x7f54e8155950 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f54e000ec50 tx=0x7f54e0009250 comp rx=0 tx=0).stop 2023-10-06T01:41:50.759 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.758+0000 7f54ed0d0700 1 -- 172.21.15.27:0/340235093 shutdown_connections 2023-10-06T01:41:50.759 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.758+0000 7f54ed0d0700 1 --2- 172.21.15.27:0/340235093 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f54c8066f30 0x7f54c80693e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:50.759 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.758+0000 7f54ed0d0700 1 --2- 172.21.15.27:0/340235093 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f54e815d930 0x7f54e8155950 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:50.760 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.758+0000 7f54ed0d0700 1 --2- 172.21.15.27:0/340235093 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f54e81591b0 0x7f54e80aab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:50.760 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.758+0000 7f54ed0d0700 1 --2- 172.21.15.27:0/340235093 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f54e815d560 0x7f54e816a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:50.760 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.758+0000 7f54ed0d0700 1 -- 172.21.15.27:0/340235093 >> 172.21.15.27:0/340235093 conn(0x7f54e80d1180 msgr2=0x7f54e8156d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:50.760 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.758+0000 7f54ed0d0700 1 -- 172.21.15.27:0/340235093 shutdown_connections 2023-10-06T01:41:50.760 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.758+0000 7f54ed0d0700 1 -- 172.21.15.27:0/340235093 wait complete. 2023-10-06T01:41:50.760 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 7 2023-10-06T01:41:50.771 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T01:41:50.889 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.889+0000 7f39af58b700 1 Processor -- start 2023-10-06T01:41:50.890 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.889+0000 7f39af58b700 1 -- start start 2023-10-06T01:41:50.892 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.890+0000 7f39af58b700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f39a80ae260 0x7f39a80ae630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:50.892 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.890+0000 7f39af58b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f39a8156f00 0x7f39a8157320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:50.892 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.890+0000 7f39af58b700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a806e010 con 0x7f39a806a150 2023-10-06T01:41:50.892 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.890+0000 7f39af58b700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39a80c0fa0 con 0x7f39a8156f00 2023-10-06T01:41:50.892 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.890+0000 7f39af58b700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39a806a5f0 con 0x7f39a80ae260 2023-10-06T01:41:50.892 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.890+0000 7f39aed8a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f39a8156f00 0x7f39a8157320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:50.893 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.890+0000 7f39aed8a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f39a8156f00 0x7f39a8157320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39546/0 (socket says 172.21.15.27:39546) 2023-10-06T01:41:50.893 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.890+0000 7f39aed8a700 1 -- 172.21.15.27:0/3570069573 learned_addr learned my addr 172.21.15.27:0/3570069573 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:50.893 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.890+0000 7f39ae589700 1 --2- 172.21.15.27:0/3570069573 >> v2:172.21.15.124:3300/0 conn(0x7f39a80ae260 0x7f39a80ae630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:50.893 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.890+0000 7f39aed8a700 1 -- 172.21.15.27:0/3570069573 >> v2:172.21.15.124:3300/0 conn(0x7f39a80ae260 msgr2=0x7f39a80ae630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:50.893 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.890+0000 7f39aed8a700 1 --2- 172.21.15.27:0/3570069573 >> v2:172.21.15.124:3300/0 conn(0x7f39a80ae260 0x7f39a80ae630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:50.893 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.890+0000 7f39aed8a700 1 -- 172.21.15.27:0/3570069573 >> v1:172.21.15.124:6789/0 conn(0x7f39a806a150 legacy=0x7f39a806d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:50.893 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.891+0000 7f39aed8a700 1 -- 172.21.15.27:0/3570069573 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a806a8d0 con 0x7f39a8156f00 2023-10-06T01:41:50.893 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.891+0000 7f39aed8a700 1 --2- 172.21.15.27:0/3570069573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39a8156f00 0x7f39a8157320 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f39a8064a60 tx=0x7f3998009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11806731c3ef904 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:50.894 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.891+0000 7f39ad587700 1 -- 172.21.15.27:0/3570069573 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399800e070 con 0x7f39a8156f00 2023-10-06T01:41:50.894 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.891+0000 7f39ad587700 1 -- 172.21.15.27:0/3570069573 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f399800ba20 con 0x7f39a8156f00 2023-10-06T01:41:50.894 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.891+0000 7f39ad587700 1 -- 172.21.15.27:0/3570069573 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39980092e0 con 0x7f39a8156f00 2023-10-06T01:41:50.894 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.891+0000 7f39af58b700 1 -- 172.21.15.27:0/3570069573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39a8156f00 msgr2=0x7f39a8157320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:50.894 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.891+0000 7f39af58b700 1 --2- 172.21.15.27:0/3570069573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39a8156f00 0x7f39a8157320 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f39a8064a60 tx=0x7f3998009840 comp rx=0 tx=0).stop 2023-10-06T01:41:50.894 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.891+0000 7f39af58b700 1 -- 172.21.15.27:0/3570069573 shutdown_connections 2023-10-06T01:41:50.894 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.891+0000 7f39af58b700 1 --2- 172.21.15.27:0/3570069573 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39a8156f00 0x7f39a8157320 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:50.895 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.891+0000 7f39af58b700 1 --2- 172.21.15.27:0/3570069573 >> v2:172.21.15.124:3300/0 conn(0x7f39a80ae260 0x7f39a80ae630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:50.895 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.891+0000 7f39af58b700 1 -- 172.21.15.27:0/3570069573 >> 172.21.15.27:0/3570069573 conn(0x7f39a80c95d0 msgr2=0x7f39a8067630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:50.895 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.892+0000 7f39af58b700 1 -- 172.21.15.27:0/3570069573 shutdown_connections 2023-10-06T01:41:50.895 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.892+0000 7f39af58b700 1 -- 172.21.15.27:0/3570069573 wait complete. 2023-10-06T01:41:50.901 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.893+0000 7f39af58b700 1 Processor -- start 2023-10-06T01:41:50.901 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.893+0000 7f39af58b700 1 -- start start 2023-10-06T01:41:50.901 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.893+0000 7f39af58b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39a806a150 0x7f39a814fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:50.901 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.893+0000 7f39af58b700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f39a80ae260 0x7f39a80af940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:50.901 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.893+0000 7f39af58b700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f39a8156f00 0x7f39a80c42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:50.902 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.893+0000 7f39af58b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a80c3240 con 0x7f39a806a150 2023-10-06T01:41:50.902 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.893+0000 7f39af58b700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a80c3520 con 0x7f39a80ae260 2023-10-06T01:41:50.902 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.893+0000 7f39af58b700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39a80c3800 con 0x7f39a8156f00 2023-10-06T01:41:50.902 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.893+0000 7f39ae589700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39a806a150 0x7f39a814fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:50.902 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.894+0000 7f39ae589700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39a806a150 0x7f39a814fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39548/0 (socket says 172.21.15.27:39548) 2023-10-06T01:41:50.902 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.894+0000 7f39ae589700 1 -- 172.21.15.27:0/541545555 learned_addr learned my addr 172.21.15.27:0/541545555 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:50.903 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.894+0000 7f39add88700 1 --2- 172.21.15.27:0/541545555 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f39a80ae260 0x7f39a80af940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:50.903 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.894+0000 7f39aed8a700 1 --2- 172.21.15.27:0/541545555 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f39a8156f00 0x7f39a80c42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:50.903 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.894+0000 7f39ae589700 1 -- 172.21.15.27:0/541545555 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f39a8156f00 msgr2=0x7f39a80c42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:50.903 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.894+0000 7f39ae589700 1 --2- 172.21.15.27:0/541545555 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f39a8156f00 0x7f39a80c42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:50.903 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.894+0000 7f39ae589700 1 -- 172.21.15.27:0/541545555 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f39a80ae260 msgr2=0x7f39a80af940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:50.903 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.894+0000 7f39ae589700 1 --2- 172.21.15.27:0/541545555 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f39a80ae260 0x7f39a80af940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:50.904 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.894+0000 7f39ae589700 1 -- 172.21.15.27:0/541545555 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3998009ae0 con 0x7f39a806a150 2023-10-06T01:41:50.904 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.894+0000 7f39aed8a700 1 --2- 172.21.15.27:0/541545555 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f39a8156f00 0x7f39a80c42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:50.904 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.894+0000 7f39ae589700 1 --2- 172.21.15.27:0/541545555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39a806a150 0x7f39a814fac0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f39a000ec80 tx=0x7f39a00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:50.904 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.894+0000 7f39a67fc700 1 -- 172.21.15.27:0/541545555 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a000c850 con 0x7f39a806a150 2023-10-06T01:41:50.904 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.894+0000 7f39a67fc700 1 -- 172.21.15.27:0/541545555 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39a000c9b0 con 0x7f39a806a150 2023-10-06T01:41:50.904 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.894+0000 7f39af58b700 1 -- 172.21.15.27:0/541545555 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39a80c3ae0 con 0x7f39a806a150 2023-10-06T01:41:50.904 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.894+0000 7f39a67fc700 1 -- 172.21.15.27:0/541545555 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a000cd50 con 0x7f39a806a150 2023-10-06T01:41:50.905 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.894+0000 7f39af58b700 1 -- 172.21.15.27:0/541545555 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39a8150000 con 0x7f39a806a150 2023-10-06T01:41:50.905 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.895+0000 7f39af58b700 1 -- 172.21.15.27:0/541545555 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39a804c3c0 con 0x7f39a806a150 2023-10-06T01:41:50.905 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.896+0000 7f39a67fc700 1 -- 172.21.15.27:0/541545555 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f39a0013680 con 0x7f39a806a150 2023-10-06T01:41:50.905 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.896+0000 7f39a67fc700 1 --2- 172.21.15.27:0/541545555 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3994066f60 0x7f3994069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:50.905 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.896+0000 7f39a67fc700 1 -- 172.21.15.27:0/541545555 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f39a0084930 con 0x7f39a806a150 2023-10-06T01:41:50.905 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.896+0000 7f39add88700 1 --2- 172.21.15.27:0/541545555 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3994066f60 0x7f3994069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:50.906 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.897+0000 7f39add88700 1 --2- 172.21.15.27:0/541545555 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3994066f60 0x7f3994069410 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f399c003a80 tx=0x7f399c007ec0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:50.906 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:50.900+0000 7f39a67fc700 1 -- 172.21.15.27:0/541545555 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f39a0051a40 con 0x7f39a806a150 2023-10-06T01:41:51.095 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.094+0000 7f39af58b700 1 -- 172.21.15.27:0/541545555 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39a8069060 con 0x7f39a806a150 2023-10-06T01:41:51.095 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.095+0000 7f39a67fc700 1 -- 172.21.15.27:0/541545555 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16866 (secure 0 0 0) 0x7f39a0055060 con 0x7f39a806a150 2023-10-06T01:41:51.096 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:51.096 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":20,"fsid":"20a7bff0-74f8-4f2f-b6c6-899e804bc965","created":"2023-10-06T01:40:54.474215+0000","modified":"2023-10-06T01:41:14.324546+0000","last_up_change":"2023-10-06T01:41:05.242847+0000","last_in_change":"2023-10-06T01:40:56.890960+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-10-06T01:41:05.685386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-10-06T01:41:06.092995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T01:41:10.434257+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T01:41:11.700262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b902c86d-800c-40c5-b152-b545e50d02c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":90644},{"type":"v1","addr":"172.21.15.27:6801","nonce":90644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":90644},{"type":"v1","addr":"172.21.15.27:6803","nonce":90644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":90644},{"type":"v1","addr":"172.21.15.27:6807","nonce":90644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":90644},{"type":"v1","addr":"172.21.15.27:6805","nonce":90644}]},"public_addr":"172.21.15.27:6801/90644","cluster_addr":"172.21.15.27:6803/90644","heartbeat_back_addr":"172.21.15.27:6807/90644","heartbeat_front_addr":"172.21.15.27:6805/90644","state":["exists","up"]},{"osd":1,"uuid":"5b4a70e5-b06e-48cd-bfc4-7962c5d05386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":90643},{"type":"v1","addr":"172.21.15.27:6809","nonce":90643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":90643},{"type":"v1","addr":"172.21.15.27:6811","nonce":90643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":90643},{"type":"v1","addr":"172.21.15.27:6815","nonce":90643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":90643},{"type":"v1","addr":"172.21.15.27:6813","nonce":90643}]},"public_addr":"172.21.15.27:6809/90643","cluster_addr":"172.21.15.27:6811/90643","heartbeat_back_addr":"172.21.15.27:6815/90643","heartbeat_front_addr":"172.21.15.27:6813/90643","state":["exists","up"]},{"osd":2,"uuid":"22d9de46-4073-4336-9122-5227e601eca0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":90645},{"type":"v1","addr":"172.21.15.27:6817","nonce":90645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":90645},{"type":"v1","addr":"172.21.15.27:6819","nonce":90645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":90645},{"type":"v1","addr":"172.21.15.27:6823","nonce":90645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":90645},{"type":"v1","addr":"172.21.15.27:6821","nonce":90645}]},"public_addr":"172.21.15.27:6817/90645","cluster_addr":"172.21.15.27:6819/90645","heartbeat_back_addr":"172.21.15.27:6823/90645","heartbeat_front_addr":"172.21.15.27:6821/90645","state":["exists","up"]},{"osd":3,"uuid":"438bb2d3-34e2-4b7c-9734-8360c1b67650","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":90668},{"type":"v1","addr":"172.21.15.27:6825","nonce":90668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":90668},{"type":"v1","addr":"172.21.15.27:6827","nonce":90668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":90668},{"type":"v1","addr":"172.21.15.27:6831","nonce":90668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":90668},{"type":"v1","addr":"172.21.15.27:6829","nonce":90668}]},"public_addr":"172.21.15.27:6825/90668","cluster_addr":"172.21.15.27:6827/90668","heartbeat_back_addr":"172.21.15.27:6831/90668","heartbeat_front_addr":"172.21.15.27:6829/90668","state":["exists","up"]},{"osd":4,"uuid":"e1956a8e-e0f8-4e7c-acbb-40037415a354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":90863},{"type":"v1","addr":"172.21.15.124:6801","nonce":90863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":90863},{"type":"v1","addr":"172.21.15.124:6803","nonce":90863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":90863},{"type":"v1","addr":"172.21.15.124:6807","nonce":90863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":90863},{"type":"v1","addr":"172.21.15.124:6805","nonce":90863}]},"public_addr":"172.21.15.124:6801/90863","cluster_addr":"172.21.15.124:6803/90863","heartbeat_back_addr":"172.21.15.124:6807/90863","heartbeat_front_addr":"172.21.15.124:6805/90863","state":["exists","up"]},{"osd":5,"uuid":"306c59d4-846d-4b73-b1c9-051bec9e9368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":90885},{"type":"v1","addr":"172.21.15.124:6817","nonce":90885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":90885},{"type":"v1","addr":"172.21.15.124:6819","nonce":90885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":90885},{"type":"v1","addr":"172.21.15.124:6823","nonce":90885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":90885},{"type":"v1","addr":"172.21.15.124:6821","nonce":90885}]},"public_addr":"172.21.15.124:6817/90885","cluster_addr":"172.21.15.124:6819/90885","heartbeat_back_addr":"172.21.15.124:6823/90885","heartbeat_front_addr":"172.21.15.124:6821/90885","state":["exists","up"]},{"osd":6,"uuid":"15463e53-ac8d-47d1-b4f6-c0d31f29e4a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":90864},{"type":"v1","addr":"172.21.15.124:6809","nonce":90864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":90864},{"type":"v1","addr":"172.21.15.124:6811","nonce":90864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":90864},{"type":"v1","addr":"172.21.15.124:6815","nonce":90864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":90864},{"type":"v1","addr":"172.21.15.124:6813","nonce":90864}]},"public_addr":"172.21.15.124:6809/90864","cluster_addr":"172.21.15.124:6811/90864","heartbeat_back_addr":"172.21.15.124:6815/90864","heartbeat_front_addr":"172.21.15.124:6813/90864","state":["exists","up"]},{"osd":7,"uuid":"ae463a90-a9d3-498f-86fa-47a35b3fe889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":90894},{"type":"v1","addr":"172.21.15.124:6825","nonce":90894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":90894},{"type":"v1","addr":"172.21.15.124:6827","nonce":90894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":90894},{"type":"v1","addr":"172.21.15.124:6831","nonce":90894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":90894},{"type":"v1","addr":"172.21.15.124:6829","nonce":90894}]},"public_addr":"172.21.15.124:6825/90894","cluster_addr":"172.21.15.124:6827/90894","heartbeat_back_addr":"172.21.15.124:6831/90894","heartbeat_front_addr":"172.21.15.124:6829/90894","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.827704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.839951+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.032230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.050605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.987908+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.031290+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.080707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.053692+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T01:41:51.101 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.097+0000 7f39af58b700 1 -- 172.21.15.27:0/541545555 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3994066f60 msgr2=0x7f3994069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:51.101 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.097+0000 7f39af58b700 1 --2- 172.21.15.27:0/541545555 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3994066f60 0x7f3994069410 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f399c003a80 tx=0x7f399c007ec0 comp rx=0 tx=0).stop 2023-10-06T01:41:51.101 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.097+0000 7f39af58b700 1 -- 172.21.15.27:0/541545555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39a806a150 msgr2=0x7f39a814fac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:51.101 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.097+0000 7f39af58b700 1 --2- 172.21.15.27:0/541545555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39a806a150 0x7f39a814fac0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f39a000ec80 tx=0x7f39a00091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:51.101 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.097+0000 7f39af58b700 1 -- 172.21.15.27:0/541545555 shutdown_connections 2023-10-06T01:41:51.101 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.097+0000 7f39af58b700 1 --2- 172.21.15.27:0/541545555 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3994066f60 0x7f3994069410 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:51.102 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.097+0000 7f39af58b700 1 --2- 172.21.15.27:0/541545555 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f39a8156f00 0x7f39a80c42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:51.102 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.097+0000 7f39af58b700 1 --2- 172.21.15.27:0/541545555 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39a806a150 0x7f39a814fac0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:51.102 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.097+0000 7f39af58b700 1 --2- 172.21.15.27:0/541545555 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f39a80ae260 0x7f39a80af940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:51.102 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.097+0000 7f39af58b700 1 -- 172.21.15.27:0/541545555 >> 172.21.15.27:0/541545555 conn(0x7f39a80c95d0 msgr2=0x7f39a806d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:51.102 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.098+0000 7f39af58b700 1 -- 172.21.15.27:0/541545555 shutdown_connections 2023-10-06T01:41:51.102 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.098+0000 7f39af58b700 1 -- 172.21.15.27:0/541545555 wait complete. 2023-10-06T01:41:51.110 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-06T01:41:51.110 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T01:41:51.229 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.229+0000 7f3d74f43700 1 Processor -- start 2023-10-06T01:41:51.230 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.229+0000 7f3d74f43700 1 -- start start 2023-10-06T01:41:51.232 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.230+0000 7f3d74f43700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3d700b0310 0x7f3d700b0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:51.232 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.230+0000 7f3d74f43700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d70069f70 con 0x7f3d7015d850 2023-10-06T01:41:51.232 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.230+0000 7f3d74f43700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d700b0c90 con 0x7f3d7015d480 2023-10-06T01:41:51.232 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.230+0000 7f3d74f43700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d700687c0 con 0x7f3d700b0310 2023-10-06T01:41:51.232 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.230+0000 7f3d6effd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f3d7015d850 0x7f3d700b5c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38510/0 (socket says 172.21.15.27:38510) 2023-10-06T01:41:51.232 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.230+0000 7f3d6effd700 1 -- 172.21.15.27:0/3137577636 learned_addr learned my addr 172.21.15.27:0/3137577636 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:51.232 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.230+0000 7f3d6ffff700 1 --2- 172.21.15.27:0/3137577636 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3d700b0310 0x7f3d700b0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:51.232 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.230+0000 7f3d6e7fc700 1 -- 172.21.15.27:0/3137577636 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2568786173 0 0) 0x7f3d700b0c90 con 0x7f3d7015d480 2023-10-06T01:41:51.232 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.230+0000 7f3d6e7fc700 1 -- 172.21.15.27:0/3137577636 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d58003570 con 0x7f3d7015d480 2023-10-06T01:41:51.232 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.230+0000 7f3d6e7fc700 1 -- 172.21.15.27:0/3137577636 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 893475479 0 0) 0x7f3d70069f70 con 0x7f3d7015d850 2023-10-06T01:41:51.233 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.230+0000 7f3d6e7fc700 1 -- 172.21.15.27:0/3137577636 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d700b0c90 con 0x7f3d7015d850 2023-10-06T01:41:51.233 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.230+0000 7f3d6ffff700 1 -- 172.21.15.27:0/3137577636 >> v1:172.21.15.124:6789/0 conn(0x7f3d7015d480 legacy=0x7f3d70069860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:51.233 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.230+0000 7f3d6ffff700 1 -- 172.21.15.27:0/3137577636 >> v1:172.21.15.27:6789/0 conn(0x7f3d7015d850 legacy=0x7f3d700b5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:51.233 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.230+0000 7f3d6ffff700 1 -- 172.21.15.27:0/3137577636 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d70068aa0 con 0x7f3d700b0310 2023-10-06T01:41:51.233 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.231+0000 7f3d6ffff700 1 --2- 172.21.15.27:0/3137577636 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3d700b0310 0x7f3d700b0750 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3d68009e90 tx=0x7f3d68009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bfa1c5ff2d2a848 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:51.233 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.231+0000 7f3d6e7fc700 1 -- 172.21.15.27:0/3137577636 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6800e070 con 0x7f3d700b0310 2023-10-06T01:41:51.233 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.231+0000 7f3d6e7fc700 1 -- 172.21.15.27:0/3137577636 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d6800bb80 con 0x7f3d700b0310 2023-10-06T01:41:51.233 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.231+0000 7f3d6e7fc700 1 -- 172.21.15.27:0/3137577636 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d68012430 con 0x7f3d700b0310 2023-10-06T01:41:51.233 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.231+0000 7f3d74f43700 1 -- 172.21.15.27:0/3137577636 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3d700b0310 msgr2=0x7f3d700b0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:51.233 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.231+0000 7f3d74f43700 1 --2- 172.21.15.27:0/3137577636 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3d700b0310 0x7f3d700b0750 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3d68009e90 tx=0x7f3d68009780 comp rx=0 tx=0).stop 2023-10-06T01:41:51.233 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.231+0000 7f3d74f43700 1 -- 172.21.15.27:0/3137577636 shutdown_connections 2023-10-06T01:41:51.233 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.231+0000 7f3d74f43700 1 --2- 172.21.15.27:0/3137577636 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3d700b0310 0x7f3d700b0750 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:51.234 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.231+0000 7f3d74f43700 1 -- 172.21.15.27:0/3137577636 >> 172.21.15.27:0/3137577636 conn(0x7f3d700ccf80 msgr2=0x7f3d700cd380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:51.234 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.231+0000 7f3d74f43700 1 -- 172.21.15.27:0/3137577636 shutdown_connections 2023-10-06T01:41:51.234 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.231+0000 7f3d74f43700 1 -- 172.21.15.27:0/3137577636 wait complete. 2023-10-06T01:41:51.236 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.233+0000 7f3d74f43700 1 Processor -- start 2023-10-06T01:41:51.236 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.233+0000 7f3d74f43700 1 -- start start 2023-10-06T01:41:51.236 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.233+0000 7f3d74f43700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3d700b0310 0x7f3d7016d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:51.236 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.233+0000 7f3d74f43700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d7015d480 0x7f3d701697c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:51.236 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.233+0000 7f3d74f43700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3d7015d850 0x7f3d70140490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:51.236 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.233+0000 7f3d74f43700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7016ddc0 con 0x7f3d7015d480 2023-10-06T01:41:51.237 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.233+0000 7f3d74f43700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7016a820 con 0x7f3d7015d850 2023-10-06T01:41:51.237 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.233+0000 7f3d74f43700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d700c8110 con 0x7f3d700b0310 2023-10-06T01:41:51.237 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.233+0000 7f3d6effd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d7015d480 0x7f3d701697c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:51.237 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.234+0000 7f3d6effd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d7015d480 0x7f3d701697c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39550/0 (socket says 172.21.15.27:39550) 2023-10-06T01:41:51.237 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.234+0000 7f3d6effd700 1 -- 172.21.15.27:0/108448015 learned_addr learned my addr 172.21.15.27:0/108448015 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:51.237 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.234+0000 7f3d6ffff700 1 --2- 172.21.15.27:0/108448015 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3d7015d850 0x7f3d70140490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:51.237 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.234+0000 7f3d6f7fe700 1 --2- 172.21.15.27:0/108448015 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3d700b0310 0x7f3d7016d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:51.237 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.234+0000 7f3d6effd700 1 -- 172.21.15.27:0/108448015 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3d700b0310 msgr2=0x7f3d7016d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:51.237 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.234+0000 7f3d6effd700 1 --2- 172.21.15.27:0/108448015 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3d700b0310 0x7f3d7016d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:51.237 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.234+0000 7f3d6effd700 1 -- 172.21.15.27:0/108448015 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3d7015d850 msgr2=0x7f3d70140490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:51.237 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.234+0000 7f3d6effd700 1 --2- 172.21.15.27:0/108448015 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3d7015d850 0x7f3d70140490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:51.237 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.234+0000 7f3d6effd700 1 -- 172.21.15.27:0/108448015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d68009a20 con 0x7f3d7015d480 2023-10-06T01:41:51.238 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.234+0000 7f3d6ffff700 1 --2- 172.21.15.27:0/108448015 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3d7015d850 0x7f3d70140490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:41:51.238 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.234+0000 7f3d6effd700 1 --2- 172.21.15.27:0/108448015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d7015d480 0x7f3d701697c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3d6400c590 tx=0x7f3d64007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:51.238 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.234+0000 7f3d577fe700 1 -- 172.21.15.27:0/108448015 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d64005b30 con 0x7f3d7015d480 2023-10-06T01:41:51.238 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.234+0000 7f3d577fe700 1 -- 172.21.15.27:0/108448015 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d64005cb0 con 0x7f3d7015d480 2023-10-06T01:41:51.238 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.234+0000 7f3d74f43700 1 -- 172.21.15.27:0/108448015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d70169d00 con 0x7f3d7015d480 2023-10-06T01:41:51.238 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.234+0000 7f3d577fe700 1 -- 172.21.15.27:0/108448015 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d64007320 con 0x7f3d7015d480 2023-10-06T01:41:51.238 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.234+0000 7f3d74f43700 1 -- 172.21.15.27:0/108448015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d7016a260 con 0x7f3d7015d480 2023-10-06T01:41:51.238 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.235+0000 7f3d74f43700 1 -- 172.21.15.27:0/108448015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d7004c3c0 con 0x7f3d7015d480 2023-10-06T01:41:51.238 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.236+0000 7f3d577fe700 1 -- 172.21.15.27:0/108448015 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3d6401b3b0 con 0x7f3d7015d480 2023-10-06T01:41:51.238 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.236+0000 7f3d577fe700 1 --2- 172.21.15.27:0/108448015 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3d5c066f30 0x7f3d5c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:51.238 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.236+0000 7f3d577fe700 1 -- 172.21.15.27:0/108448015 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f3d64083b40 con 0x7f3d7015d480 2023-10-06T01:41:51.238 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.236+0000 7f3d6f7fe700 1 --2- 172.21.15.27:0/108448015 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3d5c066f30 0x7f3d5c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:51.239 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.237+0000 7f3d6f7fe700 1 --2- 172.21.15.27:0/108448015 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3d5c066f30 0x7f3d5c0693e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3d60007fd0 tx=0x7f3d60007d50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:51.240 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.240+0000 7f3d577fe700 1 -- 172.21.15.27:0/108448015 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3d64050c50 con 0x7f3d7015d480 2023-10-06T01:41:51.477 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.477+0000 7f3d74f43700 1 -- 172.21.15.27:0/108448015 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d700c8250 con 0x7f3d7015d480 2023-10-06T01:41:51.478 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.477+0000 7f3d577fe700 1 -- 172.21.15.27:0/108448015 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4594 (secure 0 0 0) 0x7f3d64017020 con 0x7f3d7015d480 2023-10-06T01:41:51.478 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:51.478 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4261,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6837/2699906167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6836","nonce":2699906167},{"type":"v1","addr":"172.21.15.124:6837","nonce":2699906167}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/313372271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":313372271},{"type":"v1","addr":"172.21.15.27:6834","nonce":313372271}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4287,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/4257743842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4257743842},{"type":"v1","addr":"172.21.15.27:6835","nonce":4257743842}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T01:41:12.735970+0000","modified":"2023-10-06T01:41:16.358924+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.124:6835/3803994994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":3803994994},{"type":"v1","addr":"172.21.15.124:6835","nonce":3803994994}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-06T01:41:51.479 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.479+0000 7f3d74f43700 1 -- 172.21.15.27:0/108448015 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3d5c066f30 msgr2=0x7f3d5c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:51.480 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.479+0000 7f3d74f43700 1 --2- 172.21.15.27:0/108448015 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3d5c066f30 0x7f3d5c0693e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3d60007fd0 tx=0x7f3d60007d50 comp rx=0 tx=0).stop 2023-10-06T01:41:51.480 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.479+0000 7f3d74f43700 1 -- 172.21.15.27:0/108448015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d7015d480 msgr2=0x7f3d701697c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:51.480 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.479+0000 7f3d74f43700 1 --2- 172.21.15.27:0/108448015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d7015d480 0x7f3d701697c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3d6400c590 tx=0x7f3d64007e00 comp rx=0 tx=0).stop 2023-10-06T01:41:51.480 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.479+0000 7f3d74f43700 1 -- 172.21.15.27:0/108448015 shutdown_connections 2023-10-06T01:41:51.480 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.479+0000 7f3d74f43700 1 --2- 172.21.15.27:0/108448015 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3d5c066f30 0x7f3d5c0693e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:51.480 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.479+0000 7f3d74f43700 1 --2- 172.21.15.27:0/108448015 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3d700b0310 0x7f3d7016d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:51.480 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.479+0000 7f3d74f43700 1 --2- 172.21.15.27:0/108448015 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3d7015d850 0x7f3d70140490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:51.480 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.479+0000 7f3d74f43700 1 --2- 172.21.15.27:0/108448015 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3d7015d480 0x7f3d701697c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:51.480 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.479+0000 7f3d74f43700 1 -- 172.21.15.27:0/108448015 >> 172.21.15.27:0/108448015 conn(0x7f3d700ccf80 msgr2=0x7f3d700697f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:51.480 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.479+0000 7f3d74f43700 1 -- 172.21.15.27:0/108448015 shutdown_connections 2023-10-06T01:41:51.481 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.479+0000 7f3d74f43700 1 -- 172.21.15.27:0/108448015 wait complete. 2023-10-06T01:41:51.481 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 7 2023-10-06T01:41:51.492 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T01:41:51.610 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.610+0000 7fc619107700 1 Processor -- start 2023-10-06T01:41:51.611 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.610+0000 7fc619107700 1 -- start start 2023-10-06T01:41:51.612 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.611+0000 7fc619107700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fc614143160 0x7fc6140c43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:51.612 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.611+0000 7fc619107700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc614069f50 con 0x7fc614142d90 2023-10-06T01:41:51.612 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.611+0000 7fc619107700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc614069340 con 0x7fc6140c4900 2023-10-06T01:41:51.612 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.611+0000 7fc619107700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc614069620 con 0x7fc614143160 2023-10-06T01:41:51.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.611+0000 7fc612d9d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc614142d90 0x7fc61406a3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38516/0 (socket says 172.21.15.27:38516) 2023-10-06T01:41:51.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.611+0000 7fc612d9d700 1 -- 172.21.15.27:0/1804976926 learned_addr learned my addr 172.21.15.27:0/1804976926 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:51.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.611+0000 7fc61259c700 1 --2- 172.21.15.27:0/1804976926 >> v2:172.21.15.124:3300/0 conn(0x7fc614143160 0x7fc6140c43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:51.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.612+0000 7fc611d9b700 1 -- 172.21.15.27:0/1804976926 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2556264173 0 0) 0x7fc614069f50 con 0x7fc614142d90 2023-10-06T01:41:51.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.612+0000 7fc611d9b700 1 -- 172.21.15.27:0/1804976926 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc600003570 con 0x7fc614142d90 2023-10-06T01:41:51.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.612+0000 7fc61259c700 1 -- 172.21.15.27:0/1804976926 >> v1:172.21.15.124:6789/0 conn(0x7fc6140c4900 legacy=0x7fc6140cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:51.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.612+0000 7fc611d9b700 1 -- 172.21.15.27:0/1804976926 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1975596448 0 0) 0x7fc614069340 con 0x7fc6140c4900 2023-10-06T01:41:51.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.612+0000 7fc61259c700 1 -- 172.21.15.27:0/1804976926 >> v1:172.21.15.27:6789/0 conn(0x7fc614142d90 legacy=0x7fc61406a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:51.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.612+0000 7fc61259c700 1 -- 172.21.15.27:0/1804976926 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc614069900 con 0x7fc614143160 2023-10-06T01:41:51.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.612+0000 7fc61259c700 1 --2- 172.21.15.27:0/1804976926 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc614143160 0x7fc6140c43c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc0098e0 tx=0x7fc5fc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1df6cdc3f4e177a server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:51.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.612+0000 7fc611d9b700 1 -- 172.21.15.27:0/1804976926 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc00b640 con 0x7fc614143160 2023-10-06T01:41:51.614 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.612+0000 7fc611d9b700 1 -- 172.21.15.27:0/1804976926 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5fc00b7a0 con 0x7fc614143160 2023-10-06T01:41:51.614 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.612+0000 7fc611d9b700 1 -- 172.21.15.27:0/1804976926 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc00bb40 con 0x7fc614143160 2023-10-06T01:41:51.616 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.613+0000 7fc619107700 1 -- 172.21.15.27:0/1804976926 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc614143160 msgr2=0x7fc6140c43c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:51.616 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.613+0000 7fc619107700 1 --2- 172.21.15.27:0/1804976926 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc614143160 0x7fc6140c43c0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fc5fc0098e0 tx=0x7fc5fc0091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:51.616 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.613+0000 7fc619107700 1 -- 172.21.15.27:0/1804976926 shutdown_connections 2023-10-06T01:41:51.616 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.613+0000 7fc619107700 1 --2- 172.21.15.27:0/1804976926 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc614143160 0x7fc6140c43c0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:51.616 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.613+0000 7fc619107700 1 -- 172.21.15.27:0/1804976926 >> 172.21.15.27:0/1804976926 conn(0x7fc6140ce4d0 msgr2=0x7fc6140ce8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:51.616 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.613+0000 7fc619107700 1 -- 172.21.15.27:0/1804976926 shutdown_connections 2023-10-06T01:41:51.617 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.613+0000 7fc619107700 1 -- 172.21.15.27:0/1804976926 wait complete. 2023-10-06T01:41:51.617 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.614+0000 7fc619107700 1 Processor -- start 2023-10-06T01:41:51.617 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.614+0000 7fc619107700 1 -- start start 2023-10-06T01:41:51.617 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.615+0000 7fc619107700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc6140c4900 0x7fc6140b0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:51.617 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.615+0000 7fc619107700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc614142d90 0x7fc6140c7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:51.617 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.615+0000 7fc619107700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc614143160 0x7fc61415e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:51.617 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.615+0000 7fc619107700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc61415bbc0 con 0x7fc614142d90 2023-10-06T01:41:51.617 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.615+0000 7fc619107700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6140c7d20 con 0x7fc6140c4900 2023-10-06T01:41:51.617 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.615+0000 7fc619107700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6140c8000 con 0x7fc614143160 2023-10-06T01:41:51.617 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.615+0000 7fc61259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc614142d90 0x7fc6140c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:51.617 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.615+0000 7fc61259c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc614142d90 0x7fc6140c7350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39566/0 (socket says 172.21.15.27:39566) 2023-10-06T01:41:51.617 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.615+0000 7fc61259c700 1 -- 172.21.15.27:0/3034763983 learned_addr learned my addr 172.21.15.27:0/3034763983 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:51.618 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.615+0000 7fc612d9d700 1 --2- 172.21.15.27:0/3034763983 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc6140c4900 0x7fc6140b0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:51.618 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.615+0000 7fc61359e700 1 --2- 172.21.15.27:0/3034763983 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc614143160 0x7fc61415e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:51.618 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.615+0000 7fc61259c700 1 -- 172.21.15.27:0/3034763983 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc614143160 msgr2=0x7fc61415e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:51.618 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.615+0000 7fc61259c700 1 --2- 172.21.15.27:0/3034763983 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc614143160 0x7fc61415e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:51.618 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.615+0000 7fc61259c700 1 -- 172.21.15.27:0/3034763983 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc6140c4900 msgr2=0x7fc6140b0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:51.618 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.615+0000 7fc61259c700 1 --2- 172.21.15.27:0/3034763983 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc6140c4900 0x7fc6140b0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:51.618 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.615+0000 7fc61259c700 1 -- 172.21.15.27:0/3034763983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5fc009420 con 0x7fc614142d90 2023-10-06T01:41:51.618 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.615+0000 7fc61259c700 1 --2- 172.21.15.27:0/3034763983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc614142d90 0x7fc6140c7350 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc614068380 tx=0x7fc5fc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:51.618 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.615+0000 7fc60affd700 1 -- 172.21.15.27:0/3034763983 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc01f070 con 0x7fc614142d90 2023-10-06T01:41:51.618 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.616+0000 7fc60affd700 1 -- 172.21.15.27:0/3034763983 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5fc002f70 con 0x7fc614142d90 2023-10-06T01:41:51.618 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.616+0000 7fc619107700 1 -- 172.21.15.27:0/3034763983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6140c82e0 con 0x7fc614142d90 2023-10-06T01:41:51.618 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.616+0000 7fc60affd700 1 -- 172.21.15.27:0/3034763983 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5fc00b910 con 0x7fc614142d90 2023-10-06T01:41:51.619 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.616+0000 7fc619107700 1 -- 172.21.15.27:0/3034763983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6140c88a0 con 0x7fc614142d90 2023-10-06T01:41:51.619 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.616+0000 7fc619107700 1 -- 172.21.15.27:0/3034763983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6140c8a90 con 0x7fc614142d90 2023-10-06T01:41:51.619 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.617+0000 7fc60affd700 1 -- 172.21.15.27:0/3034763983 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc5fc023070 con 0x7fc614142d90 2023-10-06T01:41:51.620 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.617+0000 7fc60affd700 1 --2- 172.21.15.27:0/3034763983 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc5f4066f30 0x7fc5f40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:51.620 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.617+0000 7fc60affd700 1 -- 172.21.15.27:0/3034763983 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fc5fc085920 con 0x7fc614142d90 2023-10-06T01:41:51.620 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.618+0000 7fc612d9d700 1 --2- 172.21.15.27:0/3034763983 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc5f4066f30 0x7fc5f40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:51.620 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.618+0000 7fc612d9d700 1 --2- 172.21.15.27:0/3034763983 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc5f4066f30 0x7fc5f40693e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc604007fd0 tx=0x7fc604009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:51.622 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.622+0000 7fc60affd700 1 -- 172.21.15.27:0/3034763983 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc5fc052a30 con 0x7fc614142d90 2023-10-06T01:41:51.859 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.858+0000 7fc619107700 1 -- 172.21.15.27:0/3034763983 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc614064050 con 0x7fc614142d90 2023-10-06T01:41:51.859 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.859+0000 7fc60affd700 1 -- 172.21.15.27:0/3034763983 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4594 (secure 0 0 0) 0x7fc5fc015090 con 0x7fc614142d90 2023-10-06T01:41:51.859 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:51.859 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4261,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6837/2699906167","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6836","nonce":2699906167},{"type":"v1","addr":"172.21.15.124:6837","nonce":2699906167}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4284,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/313372271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":313372271},{"type":"v1","addr":"172.21.15.27:6834","nonce":313372271}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4287,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/4257743842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":4257743842},{"type":"v1","addr":"172.21.15.27:6835","nonce":4257743842}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T01:41:12.735970+0000","modified":"2023-10-06T01:41:16.358924+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4305},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4305":{"gid":4305,"name":"d","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.124:6835/3803994994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":3803994994},{"type":"v1","addr":"172.21.15.124:6835","nonce":3803994994}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2023-10-06T01:41:51.862 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.861+0000 7fc619107700 1 -- 172.21.15.27:0/3034763983 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc5f4066f30 msgr2=0x7fc5f40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:51.862 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.861+0000 7fc619107700 1 --2- 172.21.15.27:0/3034763983 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc5f4066f30 0x7fc5f40693e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc604007fd0 tx=0x7fc604009040 comp rx=0 tx=0).stop 2023-10-06T01:41:51.862 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.861+0000 7fc619107700 1 -- 172.21.15.27:0/3034763983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc614142d90 msgr2=0x7fc6140c7350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:51.862 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.861+0000 7fc619107700 1 --2- 172.21.15.27:0/3034763983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc614142d90 0x7fc6140c7350 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc614068380 tx=0x7fc5fc017000 comp rx=0 tx=0).stop 2023-10-06T01:41:51.862 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.861+0000 7fc619107700 1 -- 172.21.15.27:0/3034763983 shutdown_connections 2023-10-06T01:41:51.863 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.861+0000 7fc619107700 1 --2- 172.21.15.27:0/3034763983 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc5f4066f30 0x7fc5f40693e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:51.863 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.861+0000 7fc619107700 1 --2- 172.21.15.27:0/3034763983 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc614143160 0x7fc61415e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:51.863 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.861+0000 7fc619107700 1 --2- 172.21.15.27:0/3034763983 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc6140c4900 0x7fc6140b0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:51.863 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.861+0000 7fc619107700 1 --2- 172.21.15.27:0/3034763983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc614142d90 0x7fc6140c7350 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:51.863 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.861+0000 7fc619107700 1 -- 172.21.15.27:0/3034763983 >> 172.21.15.27:0/3034763983 conn(0x7fc6140ce4d0 msgr2=0x7fc6140b2ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:51.863 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.861+0000 7fc619107700 1 -- 172.21.15.27:0/3034763983 shutdown_connections 2023-10-06T01:41:51.863 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.861+0000 7fc619107700 1 -- 172.21.15.27:0/3034763983 wait complete. 2023-10-06T01:41:51.864 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 7 2023-10-06T01:41:51.874 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T01:41:51.992 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.992+0000 7f4271cc3700 1 Processor -- start 2023-10-06T01:41:51.992 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.992+0000 7f4271cc3700 1 -- start start 2023-10-06T01:41:52.000 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.992+0000 7f4271cc3700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f426c143160 0x7f426c0c43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:52.000 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.992+0000 7f4271cc3700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f426c0c4900 0x7f426c0cc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:52.000 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.992+0000 7f4271cc3700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f426c069f50 con 0x7f426c142d90 2023-10-06T01:41:52.001 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.992+0000 7f4271cc3700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f426c069340 con 0x7f426c143160 2023-10-06T01:41:52.001 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.992+0000 7f4271cc3700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f426c069620 con 0x7f426c0c4900 2023-10-06T01:41:52.001 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.992+0000 7f426affd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f426c143160 0x7f426c0c43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:52.001 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.993+0000 7f426b7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f426c142d90 0x7f426c06a3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38530/0 (socket says 172.21.15.27:38530) 2023-10-06T01:41:52.001 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.993+0000 7f426affd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f426c143160 0x7f426c0c43c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39572/0 (socket says 172.21.15.27:39572) 2023-10-06T01:41:52.001 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.993+0000 7f426b7fe700 1 -- 172.21.15.27:0/2670867283 learned_addr learned my addr 172.21.15.27:0/2670867283 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:52.001 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.993+0000 7f426bfff700 1 --2- 172.21.15.27:0/2670867283 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f426c0c4900 0x7f426c0cc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:52.002 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.993+0000 7f426affd700 1 -- 172.21.15.27:0/2670867283 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f426c0c4900 msgr2=0x7f426c0cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:52.002 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.993+0000 7f426affd700 1 --2- 172.21.15.27:0/2670867283 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f426c0c4900 0x7f426c0cc130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:52.002 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.993+0000 7f426affd700 1 -- 172.21.15.27:0/2670867283 >> v1:172.21.15.27:6789/0 conn(0x7f426c142d90 legacy=0x7f426c06a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:52.002 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.993+0000 7f426a7fc700 1 -- 172.21.15.27:0/2670867283 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 547219337 0 0) 0x7f426c069f50 con 0x7f426c142d90 2023-10-06T01:41:52.002 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.993+0000 7f426affd700 1 -- 172.21.15.27:0/2670867283 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f426c069900 con 0x7f426c143160 2023-10-06T01:41:52.002 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.993+0000 7f426affd700 1 --2- 172.21.15.27:0/2670867283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f426c143160 0x7f426c0c43c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f42600098e0 tx=0x7f42600091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f28f84878e54012 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:52.002 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.993+0000 7f426a7fc700 1 -- 172.21.15.27:0/2670867283 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f426000b640 con 0x7f426c143160 2023-10-06T01:41:52.002 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.993+0000 7f426a7fc700 1 -- 172.21.15.27:0/2670867283 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f426000b7a0 con 0x7f426c143160 2023-10-06T01:41:52.002 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.993+0000 7f426a7fc700 1 -- 172.21.15.27:0/2670867283 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f426000bb40 con 0x7f426c143160 2023-10-06T01:41:52.002 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.994+0000 7f4271cc3700 1 -- 172.21.15.27:0/2670867283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f426c143160 msgr2=0x7f426c0c43c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:52.002 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.994+0000 7f4271cc3700 1 --2- 172.21.15.27:0/2670867283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f426c143160 0x7f426c0c43c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f42600098e0 tx=0x7f42600091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:52.003 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.994+0000 7f4271cc3700 1 -- 172.21.15.27:0/2670867283 shutdown_connections 2023-10-06T01:41:52.003 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.994+0000 7f4271cc3700 1 --2- 172.21.15.27:0/2670867283 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f426c0c4900 0x7f426c0cc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:52.003 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.994+0000 7f4271cc3700 1 --2- 172.21.15.27:0/2670867283 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f426c143160 0x7f426c0c43c0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:52.003 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.994+0000 7f4271cc3700 1 -- 172.21.15.27:0/2670867283 >> 172.21.15.27:0/2670867283 conn(0x7f426c0ce4d0 msgr2=0x7f426c0ce8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:52.003 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.994+0000 7f4271cc3700 1 -- 172.21.15.27:0/2670867283 shutdown_connections 2023-10-06T01:41:52.003 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.994+0000 7f4271cc3700 1 -- 172.21.15.27:0/2670867283 wait complete. 2023-10-06T01:41:52.003 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.995+0000 7f4271cc3700 1 Processor -- start 2023-10-06T01:41:52.003 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.995+0000 7f4271cc3700 1 -- start start 2023-10-06T01:41:52.004 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.996+0000 7f4271cc3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f426c0c4900 0x7f426c13da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:52.004 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.996+0000 7f4271cc3700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f426c142d90 0x7f426c16a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:52.004 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.996+0000 7f4271cc3700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f426c143160 0x7f426c0b1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:52.004 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.996+0000 7f4271cc3700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426c13fc10 con 0x7f426c0c4900 2023-10-06T01:41:52.004 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.996+0000 7f4271cc3700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426c16a7a0 con 0x7f426c143160 2023-10-06T01:41:52.004 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.996+0000 7f4271cc3700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f426c0c56b0 con 0x7f426c142d90 2023-10-06T01:41:52.004 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.996+0000 7f426b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f426c0c4900 0x7f426c13da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:52.004 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.996+0000 7f426b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f426c0c4900 0x7f426c13da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39586/0 (socket says 172.21.15.27:39586) 2023-10-06T01:41:52.004 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.996+0000 7f426b7fe700 1 -- 172.21.15.27:0/3531691537 learned_addr learned my addr 172.21.15.27:0/3531691537 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:52.004 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.996+0000 7f426affd700 1 --2- 172.21.15.27:0/3531691537 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f426c142d90 0x7f426c16a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:52.005 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.996+0000 7f426bfff700 1 --2- 172.21.15.27:0/3531691537 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f426c143160 0x7f426c0b1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:52.005 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.996+0000 7f426b7fe700 1 -- 172.21.15.27:0/3531691537 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f426c142d90 msgr2=0x7f426c16a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:52.005 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.996+0000 7f426b7fe700 1 --2- 172.21.15.27:0/3531691537 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f426c142d90 0x7f426c16a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:52.005 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.996+0000 7f426b7fe700 1 -- 172.21.15.27:0/3531691537 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f426c143160 msgr2=0x7f426c0b1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:52.005 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.996+0000 7f426b7fe700 1 --2- 172.21.15.27:0/3531691537 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f426c143160 0x7f426c0b1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:52.005 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.996+0000 7f426b7fe700 1 -- 172.21.15.27:0/3531691537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4260009420 con 0x7f426c0c4900 2023-10-06T01:41:52.005 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.996+0000 7f426affd700 1 --2- 172.21.15.27:0/3531691537 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f426c142d90 0x7f426c16a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:41:52.005 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.996+0000 7f426bfff700 1 --2- 172.21.15.27:0/3531691537 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f426c143160 0x7f426c0b1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:41:52.005 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.996+0000 7f426b7fe700 1 --2- 172.21.15.27:0/3531691537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f426c0c4900 0x7f426c13da90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f425c00c7e0 tx=0x7f425c007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:52.005 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.997+0000 7f42537fe700 1 -- 172.21.15.27:0/3531691537 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425c005b30 con 0x7f426c0c4900 2023-10-06T01:41:52.005 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.997+0000 7f42537fe700 1 -- 172.21.15.27:0/3531691537 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f425c005c90 con 0x7f426c0c4900 2023-10-06T01:41:52.005 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.997+0000 7f4271cc3700 1 -- 172.21.15.27:0/3531691537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f426c0c5990 con 0x7f426c0c4900 2023-10-06T01:41:52.005 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.997+0000 7f42537fe700 1 -- 172.21.15.27:0/3531691537 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425c007300 con 0x7f426c0c4900 2023-10-06T01:41:52.006 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.997+0000 7f4271cc3700 1 -- 172.21.15.27:0/3531691537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f426c1400d0 con 0x7f426c0c4900 2023-10-06T01:41:52.006 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.997+0000 7f4271cc3700 1 -- 172.21.15.27:0/3531691537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f426c1402f0 con 0x7f426c0c4900 2023-10-06T01:41:52.006 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.998+0000 7f42537fe700 1 -- 172.21.15.27:0/3531691537 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f425c01b400 con 0x7f426c0c4900 2023-10-06T01:41:52.006 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.998+0000 7f42537fe700 1 --2- 172.21.15.27:0/3531691537 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4258066f30 0x7f42580693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:52.006 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.998+0000 7f42537fe700 1 -- 172.21.15.27:0/3531691537 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f425c084180 con 0x7f426c0c4900 2023-10-06T01:41:52.006 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.998+0000 7f426affd700 1 --2- 172.21.15.27:0/3531691537 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4258066f30 0x7f42580693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:52.006 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:51.999+0000 7f426affd700 1 --2- 172.21.15.27:0/3531691537 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4258066f30 0x7f42580693e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4260003080 tx=0x7f4260017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:52.007 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.003+0000 7f42537fe700 1 -- 172.21.15.27:0/3531691537 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f425c0513c0 con 0x7f426c0c4900 2023-10-06T01:41:52.197 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.197+0000 7f4271cc3700 1 -- 172.21.15.27:0/3531691537 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f426c064050 con 0x7f426c0c4900 2023-10-06T01:41:52.198 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.198+0000 7f42537fe700 1 -- 172.21.15.27:0/3531691537 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16866 (secure 0 0 0) 0x7f425c007730 con 0x7f426c0c4900 2023-10-06T01:41:52.198 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:52.198 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":20,"fsid":"20a7bff0-74f8-4f2f-b6c6-899e804bc965","created":"2023-10-06T01:40:54.474215+0000","modified":"2023-10-06T01:41:14.324546+0000","last_up_change":"2023-10-06T01:41:05.242847+0000","last_in_change":"2023-10-06T01:40:56.890960+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-10-06T01:41:05.685386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-10-06T01:41:06.092995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-10-06T01:41:10.434257+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-10-06T01:41:11.700262+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b902c86d-800c-40c5-b152-b545e50d02c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":90644},{"type":"v1","addr":"172.21.15.27:6801","nonce":90644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":90644},{"type":"v1","addr":"172.21.15.27:6803","nonce":90644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":90644},{"type":"v1","addr":"172.21.15.27:6807","nonce":90644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":90644},{"type":"v1","addr":"172.21.15.27:6805","nonce":90644}]},"public_addr":"172.21.15.27:6801/90644","cluster_addr":"172.21.15.27:6803/90644","heartbeat_back_addr":"172.21.15.27:6807/90644","heartbeat_front_addr":"172.21.15.27:6805/90644","state":["exists","up"]},{"osd":1,"uuid":"5b4a70e5-b06e-48cd-bfc4-7962c5d05386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":90643},{"type":"v1","addr":"172.21.15.27:6809","nonce":90643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":90643},{"type":"v1","addr":"172.21.15.27:6811","nonce":90643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":90643},{"type":"v1","addr":"172.21.15.27:6815","nonce":90643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":90643},{"type":"v1","addr":"172.21.15.27:6813","nonce":90643}]},"public_addr":"172.21.15.27:6809/90643","cluster_addr":"172.21.15.27:6811/90643","heartbeat_back_addr":"172.21.15.27:6815/90643","heartbeat_front_addr":"172.21.15.27:6813/90643","state":["exists","up"]},{"osd":2,"uuid":"22d9de46-4073-4336-9122-5227e601eca0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":90645},{"type":"v1","addr":"172.21.15.27:6817","nonce":90645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":90645},{"type":"v1","addr":"172.21.15.27:6819","nonce":90645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":90645},{"type":"v1","addr":"172.21.15.27:6823","nonce":90645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":90645},{"type":"v1","addr":"172.21.15.27:6821","nonce":90645}]},"public_addr":"172.21.15.27:6817/90645","cluster_addr":"172.21.15.27:6819/90645","heartbeat_back_addr":"172.21.15.27:6823/90645","heartbeat_front_addr":"172.21.15.27:6821/90645","state":["exists","up"]},{"osd":3,"uuid":"438bb2d3-34e2-4b7c-9734-8360c1b67650","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":90668},{"type":"v1","addr":"172.21.15.27:6825","nonce":90668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":90668},{"type":"v1","addr":"172.21.15.27:6827","nonce":90668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":90668},{"type":"v1","addr":"172.21.15.27:6831","nonce":90668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":90668},{"type":"v1","addr":"172.21.15.27:6829","nonce":90668}]},"public_addr":"172.21.15.27:6825/90668","cluster_addr":"172.21.15.27:6827/90668","heartbeat_back_addr":"172.21.15.27:6831/90668","heartbeat_front_addr":"172.21.15.27:6829/90668","state":["exists","up"]},{"osd":4,"uuid":"e1956a8e-e0f8-4e7c-acbb-40037415a354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":90863},{"type":"v1","addr":"172.21.15.124:6801","nonce":90863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":90863},{"type":"v1","addr":"172.21.15.124:6803","nonce":90863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":90863},{"type":"v1","addr":"172.21.15.124:6807","nonce":90863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":90863},{"type":"v1","addr":"172.21.15.124:6805","nonce":90863}]},"public_addr":"172.21.15.124:6801/90863","cluster_addr":"172.21.15.124:6803/90863","heartbeat_back_addr":"172.21.15.124:6807/90863","heartbeat_front_addr":"172.21.15.124:6805/90863","state":["exists","up"]},{"osd":5,"uuid":"306c59d4-846d-4b73-b1c9-051bec9e9368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":90885},{"type":"v1","addr":"172.21.15.124:6817","nonce":90885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":90885},{"type":"v1","addr":"172.21.15.124:6819","nonce":90885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":90885},{"type":"v1","addr":"172.21.15.124:6823","nonce":90885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":90885},{"type":"v1","addr":"172.21.15.124:6821","nonce":90885}]},"public_addr":"172.21.15.124:6817/90885","cluster_addr":"172.21.15.124:6819/90885","heartbeat_back_addr":"172.21.15.124:6823/90885","heartbeat_front_addr":"172.21.15.124:6821/90885","state":["exists","up"]},{"osd":6,"uuid":"15463e53-ac8d-47d1-b4f6-c0d31f29e4a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":90864},{"type":"v1","addr":"172.21.15.124:6809","nonce":90864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":90864},{"type":"v1","addr":"172.21.15.124:6811","nonce":90864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":90864},{"type":"v1","addr":"172.21.15.124:6815","nonce":90864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":90864},{"type":"v1","addr":"172.21.15.124:6813","nonce":90864}]},"public_addr":"172.21.15.124:6809/90864","cluster_addr":"172.21.15.124:6811/90864","heartbeat_back_addr":"172.21.15.124:6815/90864","heartbeat_front_addr":"172.21.15.124:6813/90864","state":["exists","up"]},{"osd":7,"uuid":"ae463a90-a9d3-498f-86fa-47a35b3fe889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":90894},{"type":"v1","addr":"172.21.15.124:6825","nonce":90894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":90894},{"type":"v1","addr":"172.21.15.124:6827","nonce":90894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":90894},{"type":"v1","addr":"172.21.15.124:6831","nonce":90894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":90894},{"type":"v1","addr":"172.21.15.124:6829","nonce":90894}]},"public_addr":"172.21.15.124:6825/90894","cluster_addr":"172.21.15.124:6827/90894","heartbeat_back_addr":"172.21.15.124:6831/90894","heartbeat_front_addr":"172.21.15.124:6829/90894","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.827704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.839951+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.032230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.050605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.987908+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.031290+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.080707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.053692+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T01:41:52.200 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.200+0000 7f4271cc3700 1 -- 172.21.15.27:0/3531691537 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4258066f30 msgr2=0x7f42580693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:52.200 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.200+0000 7f4271cc3700 1 --2- 172.21.15.27:0/3531691537 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4258066f30 0x7f42580693e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4260003080 tx=0x7f4260017000 comp rx=0 tx=0).stop 2023-10-06T01:41:52.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.200+0000 7f4271cc3700 1 -- 172.21.15.27:0/3531691537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f426c0c4900 msgr2=0x7f426c13da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:52.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.200+0000 7f4271cc3700 1 --2- 172.21.15.27:0/3531691537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f426c0c4900 0x7f426c13da90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f425c00c7e0 tx=0x7f425c007df0 comp rx=0 tx=0).stop 2023-10-06T01:41:52.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.200+0000 7f4271cc3700 1 -- 172.21.15.27:0/3531691537 shutdown_connections 2023-10-06T01:41:52.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.200+0000 7f4271cc3700 1 --2- 172.21.15.27:0/3531691537 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4258066f30 0x7f42580693e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:52.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.200+0000 7f4271cc3700 1 --2- 172.21.15.27:0/3531691537 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f426c143160 0x7f426c0b1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:52.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.200+0000 7f4271cc3700 1 --2- 172.21.15.27:0/3531691537 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f426c0c4900 0x7f426c13da90 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:52.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.200+0000 7f4271cc3700 1 --2- 172.21.15.27:0/3531691537 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f426c142d90 0x7f426c16a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:52.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.200+0000 7f4271cc3700 1 -- 172.21.15.27:0/3531691537 >> 172.21.15.27:0/3531691537 conn(0x7f426c0ce4d0 msgr2=0x7f426c06c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:52.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.200+0000 7f4271cc3700 1 -- 172.21.15.27:0/3531691537 shutdown_connections 2023-10-06T01:41:52.201 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.200+0000 7f4271cc3700 1 -- 172.21.15.27:0/3531691537 wait complete. 2023-10-06T01:41:52.212 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-06T01:41:52.332 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.332+0000 7fc62ebe3700 1 Processor -- start 2023-10-06T01:41:52.332 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.332+0000 7fc62ebe3700 1 -- start start 2023-10-06T01:41:52.334 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.333+0000 7fc62ebe3700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fc628159500 0x7fc6281598d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:52.334 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.333+0000 7fc62ebe3700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc6280b3fd0 0x7fc62815d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:52.334 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.333+0000 7fc62ebe3700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc62815d8e0 0x7fc6280cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:52.334 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.333+0000 7fc62ebe3700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc62815dd20 con 0x7fc6280b3fd0 2023-10-06T01:41:52.335 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.333+0000 7fc62ebe3700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6280b1110 con 0x7fc628159500 2023-10-06T01:41:52.335 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.333+0000 7fc62ebe3700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6280b13b0 con 0x7fc62815d8e0 2023-10-06T01:41:52.335 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.333+0000 7fc61ffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc6280b3fd0 0x7fc62815d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:52.335 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.333+0000 7fc61ffff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc6280b3fd0 0x7fc62815d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39588/0 (socket says 172.21.15.27:39588) 2023-10-06T01:41:52.335 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.333+0000 7fc61ffff700 1 -- 172.21.15.27:0/1796888913 learned_addr learned my addr 172.21.15.27:0/1796888913 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:52.335 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.333+0000 7fc62c97f700 1 --2- 172.21.15.27:0/1796888913 >> v2:172.21.15.124:3300/0 conn(0x7fc628159500 0x7fc6281598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:52.335 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.333+0000 7fc62d180700 1 --2- 172.21.15.27:0/1796888913 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc62815d8e0 0x7fc6280cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:52.335 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.333+0000 7fc61ffff700 1 -- 172.21.15.27:0/1796888913 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc62815d8e0 msgr2=0x7fc6280cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:52.335 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.333+0000 7fc61ffff700 1 --2- 172.21.15.27:0/1796888913 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc62815d8e0 0x7fc6280cf6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:52.336 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.333+0000 7fc61ffff700 1 -- 172.21.15.27:0/1796888913 >> v2:172.21.15.124:3300/0 conn(0x7fc628159500 msgr2=0x7fc6281598d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:52.336 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.333+0000 7fc61ffff700 1 --2- 172.21.15.27:0/1796888913 >> v2:172.21.15.124:3300/0 conn(0x7fc628159500 0x7fc6281598d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:52.336 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.333+0000 7fc61ffff700 1 -- 172.21.15.27:0/1796888913 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6280b1690 con 0x7fc6280b3fd0 2023-10-06T01:41:52.336 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.333+0000 7fc61ffff700 1 --2- 172.21.15.27:0/1796888913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6280b3fd0 0x7fc62815d3a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc628063e00 tx=0x7fc618009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8e6d4c3319ca7ee server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:52.336 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.333+0000 7fc61f7fe700 1 -- 172.21.15.27:0/1796888913 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc61800b640 con 0x7fc6280b3fd0 2023-10-06T01:41:52.336 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.333+0000 7fc61f7fe700 1 -- 172.21.15.27:0/1796888913 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc61800b7a0 con 0x7fc6280b3fd0 2023-10-06T01:41:52.336 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.333+0000 7fc61f7fe700 1 -- 172.21.15.27:0/1796888913 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc61800bb60 con 0x7fc6280b3fd0 2023-10-06T01:41:52.336 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.334+0000 7fc62ebe3700 1 -- 172.21.15.27:0/1796888913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6280b3fd0 msgr2=0x7fc62815d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:52.336 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.334+0000 7fc62ebe3700 1 --2- 172.21.15.27:0/1796888913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6280b3fd0 0x7fc62815d3a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fc628063e00 tx=0x7fc618009250 comp rx=0 tx=0).stop 2023-10-06T01:41:52.336 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.334+0000 7fc62ebe3700 1 -- 172.21.15.27:0/1796888913 shutdown_connections 2023-10-06T01:41:52.337 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.334+0000 7fc62ebe3700 1 --2- 172.21.15.27:0/1796888913 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc62815d8e0 0x7fc6280cf6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:52.337 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.334+0000 7fc62ebe3700 1 --2- 172.21.15.27:0/1796888913 >> v2:172.21.15.124:3300/0 conn(0x7fc628159500 0x7fc6281598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:52.337 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.334+0000 7fc62ebe3700 1 --2- 172.21.15.27:0/1796888913 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6280b3fd0 0x7fc62815d3a0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:52.337 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.334+0000 7fc62ebe3700 1 -- 172.21.15.27:0/1796888913 >> 172.21.15.27:0/1796888913 conn(0x7fc628067630 msgr2=0x7fc628069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:52.337 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.334+0000 7fc62ebe3700 1 -- 172.21.15.27:0/1796888913 shutdown_connections 2023-10-06T01:41:52.337 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.334+0000 7fc62ebe3700 1 -- 172.21.15.27:0/1796888913 wait complete. 2023-10-06T01:41:52.340 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.335+0000 7fc62ebe3700 1 Processor -- start 2023-10-06T01:41:52.340 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.335+0000 7fc62ebe3700 1 -- start start 2023-10-06T01:41:52.340 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.336+0000 7fc62ebe3700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc6280b3fd0 0x7fc62814a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:52.341 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.336+0000 7fc62ebe3700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc628159500 0x7fc6280b1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:52.341 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.336+0000 7fc62ebe3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc62815d8e0 0x7fc6280b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:52.341 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.336+0000 7fc62ebe3700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc62814a7a0 con 0x7fc62815d8e0 2023-10-06T01:41:52.341 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.336+0000 7fc62ebe3700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc62815af70 con 0x7fc628159500 2023-10-06T01:41:52.341 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.336+0000 7fc62ebe3700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc62815b100 con 0x7fc6280b3fd0 2023-10-06T01:41:52.341 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.336+0000 7fc62d180700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc62815d8e0 0x7fc6280b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:52.341 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.336+0000 7fc62d180700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc62815d8e0 0x7fc6280b22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39594/0 (socket says 172.21.15.27:39594) 2023-10-06T01:41:52.341 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.336+0000 7fc62d180700 1 -- 172.21.15.27:0/2836096939 learned_addr learned my addr 172.21.15.27:0/2836096939 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:52.341 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.336+0000 7fc61ffff700 1 --2- 172.21.15.27:0/2836096939 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc628159500 0x7fc6280b1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:52.341 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.336+0000 7fc62c97f700 1 --2- 172.21.15.27:0/2836096939 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc6280b3fd0 0x7fc62814a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:52.342 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.336+0000 7fc62d180700 1 -- 172.21.15.27:0/2836096939 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc6280b3fd0 msgr2=0x7fc62814a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:52.342 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.336+0000 7fc62d180700 1 --2- 172.21.15.27:0/2836096939 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc6280b3fd0 0x7fc62814a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:52.342 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.336+0000 7fc62d180700 1 -- 172.21.15.27:0/2836096939 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc628159500 msgr2=0x7fc6280b1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:52.342 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.336+0000 7fc62d180700 1 --2- 172.21.15.27:0/2836096939 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc628159500 0x7fc6280b1d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:52.342 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.336+0000 7fc62d180700 1 -- 172.21.15.27:0/2836096939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6180094f0 con 0x7fc62815d8e0 2023-10-06T01:41:52.342 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.336+0000 7fc62c97f700 1 --2- 172.21.15.27:0/2836096939 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc6280b3fd0 0x7fc62814a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:41:52.342 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.337+0000 7fc62d180700 1 --2- 172.21.15.27:0/2836096939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc62815d8e0 0x7fc6280b22d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc620009df0 tx=0x7fc62000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:52.342 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.337+0000 7fc61cff9700 1 -- 172.21.15.27:0/2836096939 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62000cb50 con 0x7fc62815d8e0 2023-10-06T01:41:52.342 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.337+0000 7fc61cff9700 1 -- 172.21.15.27:0/2836096939 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc62000f040 con 0x7fc62815d8e0 2023-10-06T01:41:52.342 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.337+0000 7fc62ebe3700 1 -- 172.21.15.27:0/2836096939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc62814d930 con 0x7fc62815d8e0 2023-10-06T01:41:52.342 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.337+0000 7fc61cff9700 1 -- 172.21.15.27:0/2836096939 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc620014480 con 0x7fc62815d8e0 2023-10-06T01:41:52.342 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.337+0000 7fc62ebe3700 1 -- 172.21.15.27:0/2836096939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc62814def0 con 0x7fc62815d8e0 2023-10-06T01:41:52.343 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.338+0000 7fc62ebe3700 1 -- 172.21.15.27:0/2836096939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc628064050 con 0x7fc62815d8e0 2023-10-06T01:41:52.343 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.338+0000 7fc61cff9700 1 -- 172.21.15.27:0/2836096939 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc620014830 con 0x7fc62815d8e0 2023-10-06T01:41:52.343 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.339+0000 7fc61cff9700 1 --2- 172.21.15.27:0/2836096939 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc614066f30 0x7fc6140693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:52.343 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.339+0000 7fc61cff9700 1 -- 172.21.15.27:0/2836096939 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fc620085090 con 0x7fc62815d8e0 2023-10-06T01:41:52.343 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.339+0000 7fc62c97f700 1 --2- 172.21.15.27:0/2836096939 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc614066f30 0x7fc6140693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:52.343 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.339+0000 7fc62c97f700 1 --2- 172.21.15.27:0/2836096939 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc614066f30 0x7fc6140693e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc6240093f0 tx=0x7fc624008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:52.343 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.343+0000 7fc61cff9700 1 -- 172.21.15.27:0/2836096939 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc6200521a0 con 0x7fc62815d8e0 2023-10-06T01:41:52.584 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.584+0000 7fc62ebe3700 1 -- 172.21.15.27:0/2836096939 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fc62815b240 con 0x7fc62815d8e0 2023-10-06T01:41:52.584 INFO:tasks.ceph.mon.a.smithi027.stderr:2023-10-06T01:41:52.584+0000 7f4bb5c52700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-06T01:41:52.588 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.588+0000 7fc61cff9700 1 -- 172.21.15.27:0/2836096939 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v8) v1 ==== 161+0+0 (secure 0 0 0) 0x7fc6200557c0 con 0x7fc62815d8e0 2023-10-06T01:41:52.590 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.590+0000 7fc62ebe3700 1 -- 172.21.15.27:0/2836096939 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc614066f30 msgr2=0x7fc6140693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:52.590 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.590+0000 7fc62ebe3700 1 --2- 172.21.15.27:0/2836096939 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc614066f30 0x7fc6140693e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc6240093f0 tx=0x7fc624008040 comp rx=0 tx=0).stop 2023-10-06T01:41:52.590 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.590+0000 7fc62ebe3700 1 -- 172.21.15.27:0/2836096939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc62815d8e0 msgr2=0x7fc6280b22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:52.591 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.590+0000 7fc62ebe3700 1 --2- 172.21.15.27:0/2836096939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc62815d8e0 0x7fc6280b22d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc620009df0 tx=0x7fc62000b040 comp rx=0 tx=0).stop 2023-10-06T01:41:52.591 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.590+0000 7fc62ebe3700 1 -- 172.21.15.27:0/2836096939 shutdown_connections 2023-10-06T01:41:52.591 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.590+0000 7fc62ebe3700 1 --2- 172.21.15.27:0/2836096939 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc614066f30 0x7fc6140693e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:52.591 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.590+0000 7fc62ebe3700 1 --2- 172.21.15.27:0/2836096939 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc6280b3fd0 0x7fc62814a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:52.591 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.590+0000 7fc62ebe3700 1 --2- 172.21.15.27:0/2836096939 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc62815d8e0 0x7fc6280b22d0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:52.591 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.590+0000 7fc62ebe3700 1 --2- 172.21.15.27:0/2836096939 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc628159500 0x7fc6280b1d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:52.591 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.590+0000 7fc62ebe3700 1 -- 172.21.15.27:0/2836096939 >> 172.21.15.27:0/2836096939 conn(0x7fc628067630 msgr2=0x7fc6280b4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:52.591 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.590+0000 7fc62ebe3700 1 -- 172.21.15.27:0/2836096939 shutdown_connections 2023-10-06T01:41:52.591 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.590+0000 7fc62ebe3700 1 -- 172.21.15.27:0/2836096939 wait complete. 2023-10-06T01:41:52.592 INFO:tasks.ceph.mds.d.smithi124.stderr: -1811> 2023-10-06T01:41:18.376+0000 7f43112ad700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-06T01:41:52.592 INFO:teuthology.orchestra.run.smithi027.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-06T01:41:52.602 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-06T01:41:52.615 INFO:tasks.ceph.mds.d.smithi124.stdout:starting mds.d at 2023-10-06T01:41:52.724 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.724+0000 7fe178434700 1 Processor -- start 2023-10-06T01:41:52.724 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.724+0000 7fe178434700 1 -- start start 2023-10-06T01:41:52.725 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.725+0000 7fe178434700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe1701598d0 0x7fe17015d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:52.725 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.725+0000 7fe178434700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe17015d8e0 0x7fe1700cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:52.725 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.725+0000 7fe178434700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe17015dd20 con 0x7fe170159500 2023-10-06T01:41:52.725 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.725+0000 7fe178434700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1700b1110 con 0x7fe17015d8e0 2023-10-06T01:41:52.726 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.725+0000 7fe178434700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1700b13b0 con 0x7fe1701598d0 2023-10-06T01:41:52.726 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.725+0000 7fe1761d0700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe170159500 0x7fe1700b3fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38532/0 (socket says 172.21.15.27:38532) 2023-10-06T01:41:52.726 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.725+0000 7fe1761d0700 1 -- 172.21.15.27:0/1544916950 learned_addr learned my addr 172.21.15.27:0/1544916950 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:52.729 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.725+0000 7fe1759cf700 1 --2- 172.21.15.27:0/1544916950 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe1701598d0 0x7fe17015d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:52.729 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.725+0000 7fe1769d1700 1 --2- 172.21.15.27:0/1544916950 >> v2:172.21.15.124:3300/0 conn(0x7fe17015d8e0 0x7fe1700cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:52.729 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.725+0000 7fe1751ce700 1 -- 172.21.15.27:0/1544916950 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 853458785 0 0) 0x7fe17015dd20 con 0x7fe170159500 2023-10-06T01:41:52.729 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.725+0000 7fe1751ce700 1 -- 172.21.15.27:0/1544916950 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe15c003570 con 0x7fe170159500 2023-10-06T01:41:52.729 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.725+0000 7fe1751ce700 1 -- 172.21.15.27:0/1544916950 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1132634353 0 0) 0x7fe15c003570 con 0x7fe170159500 2023-10-06T01:41:52.729 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.725+0000 7fe1751ce700 1 -- 172.21.15.27:0/1544916950 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe1701598d0 msgr2=0x7fe17015d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:52.729 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.725+0000 7fe1751ce700 1 --2- 172.21.15.27:0/1544916950 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe1701598d0 0x7fe17015d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:52.729 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.725+0000 7fe1751ce700 1 -- 172.21.15.27:0/1544916950 >> v2:172.21.15.124:3300/0 conn(0x7fe17015d8e0 msgr2=0x7fe1700cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:52.729 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.725+0000 7fe1751ce700 1 --2- 172.21.15.27:0/1544916950 >> v2:172.21.15.124:3300/0 conn(0x7fe17015d8e0 0x7fe1700cf6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:52.729 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.726+0000 7fe1751ce700 1 -- 172.21.15.27:0/1544916950 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1700b1690 con 0x7fe170159500 2023-10-06T01:41:52.730 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.726+0000 7fe1751ce700 1 -- 172.21.15.27:0/1544916950 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2619078944 0 0) 0x7fe16c001470 con 0x7fe170159500 2023-10-06T01:41:52.730 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.726+0000 7fe1751ce700 1 -- 172.21.15.27:0/1544916950 >> v1:172.21.15.27:6789/0 conn(0x7fe170159500 legacy=0x7fe1700b3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:52.730 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.726+0000 7fe1751ce700 1 --2- 172.21.15.27:0/1544916950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe15c003a60 0x7fe15c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:52.730 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.726+0000 7fe1751ce700 1 --2- 172.21.15.27:0/1544916950 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe15c004810 0x7fe15c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:52.730 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.726+0000 7fe1751ce700 1 --2- 172.21.15.27:0/1544916950 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe1701598d0 0x7fe15c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:52.730 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.726+0000 7fe1751ce700 1 -- 172.21.15.27:0/1544916950 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1700b1110 con 0x7fe15c003a60 2023-10-06T01:41:52.730 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.726+0000 7fe1751ce700 1 -- 172.21.15.27:0/1544916950 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1700b13b0 con 0x7fe15c004810 2023-10-06T01:41:52.730 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.726+0000 7fe1751ce700 1 -- 172.21.15.27:0/1544916950 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe15c00ba60 con 0x7fe1701598d0 2023-10-06T01:41:52.730 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.726+0000 7fe1761d0700 1 --2- 172.21.15.27:0/1544916950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe15c003a60 0x7fe15c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:52.730 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.726+0000 7fe1759cf700 1 --2- 172.21.15.27:0/1544916950 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe15c004810 0x7fe15c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:52.730 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.726+0000 7fe1769d1700 1 --2- 172.21.15.27:0/1544916950 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe1701598d0 0x7fe15c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:52.730 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.726+0000 7fe1761d0700 1 -- 172.21.15.27:0/1544916950 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe1701598d0 msgr2=0x7fe15c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:52.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.726+0000 7fe1761d0700 1 --2- 172.21.15.27:0/1544916950 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe1701598d0 0x7fe15c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:52.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.726+0000 7fe1761d0700 1 -- 172.21.15.27:0/1544916950 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe15c004810 msgr2=0x7fe15c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:52.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.726+0000 7fe1761d0700 1 --2- 172.21.15.27:0/1544916950 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe15c004810 0x7fe15c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:52.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.726+0000 7fe1761d0700 1 -- 172.21.15.27:0/1544916950 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe15c003570 con 0x7fe15c003a60 2023-10-06T01:41:52.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.726+0000 7fe1769d1700 1 --2- 172.21.15.27:0/1544916950 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe1701598d0 0x7fe15c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:52.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.726+0000 7fe1761d0700 1 --2- 172.21.15.27:0/1544916950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe15c003a60 0x7fe15c003e70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe16c00c3f0 tx=0x7fe16c001e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96cf6e5dbd726769 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:52.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.726+0000 7fe1751ce700 1 -- 172.21.15.27:0/1544916950 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16c00cc80 con 0x7fe15c003a60 2023-10-06T01:41:52.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.726+0000 7fe1751ce700 1 -- 172.21.15.27:0/1544916950 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe16c00cde0 con 0x7fe15c003a60 2023-10-06T01:41:52.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.726+0000 7fe1751ce700 1 -- 172.21.15.27:0/1544916950 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16c005b70 con 0x7fe15c003a60 2023-10-06T01:41:52.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.727+0000 7fe178434700 1 -- 172.21.15.27:0/1544916950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe15c003a60 msgr2=0x7fe15c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:52.732 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.727+0000 7fe178434700 1 --2- 172.21.15.27:0/1544916950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe15c003a60 0x7fe15c003e70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe16c00c3f0 tx=0x7fe16c001e90 comp rx=0 tx=0).stop 2023-10-06T01:41:52.732 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.727+0000 7fe1769d1700 1 -- 172.21.15.27:0/1544916950 reap_dead start 2023-10-06T01:41:52.732 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.727+0000 7fe178434700 1 -- 172.21.15.27:0/1544916950 shutdown_connections 2023-10-06T01:41:52.732 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.727+0000 7fe178434700 1 -- 172.21.15.27:0/1544916950 >> 172.21.15.27:0/1544916950 conn(0x7fe170067630 msgr2=0x7fe170069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:52.732 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.727+0000 7fe178434700 1 -- 172.21.15.27:0/1544916950 shutdown_connections 2023-10-06T01:41:52.732 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.727+0000 7fe178434700 1 -- 172.21.15.27:0/1544916950 wait complete. 2023-10-06T01:41:52.732 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.728+0000 7fe178434700 1 Processor -- start 2023-10-06T01:41:52.732 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.728+0000 7fe178434700 1 -- start start 2023-10-06T01:41:52.734 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.729+0000 7fe178434700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe15c003a60 0x7fe1701420e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:52.734 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.729+0000 7fe178434700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe170169db0 0x7fe17016a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:52.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.729+0000 7fe178434700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe17014f290 0x7fe17014f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:52.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.729+0000 7fe178434700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17014fc40 con 0x7fe170169db0 2023-10-06T01:41:52.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.729+0000 7fe178434700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe170150560 con 0x7fe17014f290 2023-10-06T01:41:52.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.729+0000 7fe178434700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe17016a8b0 con 0x7fe15c003a60 2023-10-06T01:41:52.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.729+0000 7fe1759cf700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe170169db0 0x7fe17016a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:52.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.729+0000 7fe1759cf700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe170169db0 0x7fe17016a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39614/0 (socket says 172.21.15.27:39614) 2023-10-06T01:41:52.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.729+0000 7fe1759cf700 1 -- 172.21.15.27:0/2080632794 learned_addr learned my addr 172.21.15.27:0/2080632794 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:52.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.729+0000 7fe1769d1700 1 --2- 172.21.15.27:0/2080632794 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe17014f290 0x7fe17014f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:52.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.729+0000 7fe1761d0700 1 --2- 172.21.15.27:0/2080632794 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe15c003a60 0x7fe1701420e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:52.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.729+0000 7fe1759cf700 1 -- 172.21.15.27:0/2080632794 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe15c003a60 msgr2=0x7fe1701420e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:52.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.729+0000 7fe1759cf700 1 --2- 172.21.15.27:0/2080632794 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe15c003a60 0x7fe1701420e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:52.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.729+0000 7fe1759cf700 1 -- 172.21.15.27:0/2080632794 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe17014f290 msgr2=0x7fe17014f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:52.736 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.729+0000 7fe1759cf700 1 --2- 172.21.15.27:0/2080632794 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe17014f290 0x7fe17014f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:52.736 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.729+0000 7fe1759cf700 1 -- 172.21.15.27:0/2080632794 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe17016ab90 con 0x7fe170169db0 2023-10-06T01:41:52.736 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.729+0000 7fe1761d0700 1 --2- 172.21.15.27:0/2080632794 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe15c003a60 0x7fe1701420e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:52.736 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.729+0000 7fe1759cf700 1 --2- 172.21.15.27:0/2080632794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe170169db0 0x7fe17016a220 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe160011d40 tx=0x7fe1600091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:52.736 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.730+0000 7fe1667fc700 1 -- 172.21.15.27:0/2080632794 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe16000fa40 con 0x7fe170169db0 2023-10-06T01:41:52.736 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.730+0000 7fe1667fc700 1 -- 172.21.15.27:0/2080632794 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe16000fba0 con 0x7fe170169db0 2023-10-06T01:41:52.736 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.730+0000 7fe178434700 1 -- 172.21.15.27:0/2080632794 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1701501f0 con 0x7fe170169db0 2023-10-06T01:41:52.736 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.730+0000 7fe1667fc700 1 -- 172.21.15.27:0/2080632794 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe160014440 con 0x7fe170169db0 2023-10-06T01:41:52.736 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.730+0000 7fe178434700 1 -- 172.21.15.27:0/2080632794 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe17015ca10 con 0x7fe170169db0 2023-10-06T01:41:52.736 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.731+0000 7fe178434700 1 -- 172.21.15.27:0/2080632794 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe17016add0 con 0x7fe170169db0 2023-10-06T01:41:52.736 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.731+0000 7fe1667fc700 1 -- 172.21.15.27:0/2080632794 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe160014870 con 0x7fe170169db0 2023-10-06T01:41:52.736 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.731+0000 7fe1667fc700 1 --2- 172.21.15.27:0/2080632794 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe154066f30 0x7fe1540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:52.737 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.732+0000 7fe1667fc700 1 -- 172.21.15.27:0/2080632794 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6144+0+0 (secure 0 0 0) 0x7fe160018030 con 0x7fe170169db0 2023-10-06T01:41:52.737 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.732+0000 7fe1761d0700 1 --2- 172.21.15.27:0/2080632794 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe154066f30 0x7fe1540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:52.737 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.732+0000 7fe1761d0700 1 --2- 172.21.15.27:0/2080632794 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe154066f30 0x7fe1540693e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe16c00c310 tx=0x7fe16c015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:52.737 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.736+0000 7fe1667fc700 1 -- 172.21.15.27:0/2080632794 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe17016add0 con 0x7fe170169db0 2023-10-06T01:41:52.973 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:52.973+0000 7fe178434700 1 -- 172.21.15.27:0/2080632794 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe17016add0 con 0x7fe170169db0 2023-10-06T01:41:53.588 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.588+0000 7fe1667fc700 1 -- 172.21.15.27:0/2080632794 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v9) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe17016add0 con 0x7fe170169db0 2023-10-06T01:41:53.591 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.590+0000 7fe178434700 1 -- 172.21.15.27:0/2080632794 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe154066f30 msgr2=0x7fe1540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:53.591 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.590+0000 7fe178434700 1 --2- 172.21.15.27:0/2080632794 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe154066f30 0x7fe1540693e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fe16c00c310 tx=0x7fe16c015000 comp rx=0 tx=0).stop 2023-10-06T01:41:53.591 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.590+0000 7fe178434700 1 -- 172.21.15.27:0/2080632794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe170169db0 msgr2=0x7fe17016a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:53.591 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.590+0000 7fe178434700 1 --2- 172.21.15.27:0/2080632794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe170169db0 0x7fe17016a220 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe160011d40 tx=0x7fe1600091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:53.591 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.590+0000 7fe178434700 1 -- 172.21.15.27:0/2080632794 shutdown_connections 2023-10-06T01:41:53.591 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.590+0000 7fe178434700 1 --2- 172.21.15.27:0/2080632794 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe154066f30 0x7fe1540693e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:53.591 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.590+0000 7fe178434700 1 --2- 172.21.15.27:0/2080632794 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe15c003a60 0x7fe1701420e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:53.591 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.590+0000 7fe178434700 1 --2- 172.21.15.27:0/2080632794 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe17014f290 0x7fe17014f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:53.591 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.590+0000 7fe178434700 1 --2- 172.21.15.27:0/2080632794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe170169db0 0x7fe17016a220 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:53.591 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.590+0000 7fe178434700 1 -- 172.21.15.27:0/2080632794 >> 172.21.15.27:0/2080632794 conn(0x7fe170067630 msgr2=0x7fe17014e590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:53.592 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.590+0000 7fe178434700 1 -- 172.21.15.27:0/2080632794 shutdown_connections 2023-10-06T01:41:53.592 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.590+0000 7fe178434700 1 -- 172.21.15.27:0/2080632794 wait complete. 2023-10-06T01:41:53.604 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-06T01:41:53.730 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.730+0000 7f6bbac6d700 1 Processor -- start 2023-10-06T01:41:53.730 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.730+0000 7f6bbac6d700 1 -- start start 2023-10-06T01:41:53.732 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.730+0000 7f6bbac6d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6bb4067ec0 0x7f6bb406dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:53.732 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.730+0000 7f6bbac6d700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f6bb40c6b90 0x7f6bb4068290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:53.733 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.731+0000 7f6bbac6d700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bb4156000 con 0x7f6bb40687d0 2023-10-06T01:41:53.733 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.731+0000 7f6bbac6d700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bb406f260 con 0x7f6bb4067ec0 2023-10-06T01:41:53.733 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.731+0000 7f6bbac6d700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bb4067d40 con 0x7f6bb40c6b90 2023-10-06T01:41:53.733 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.731+0000 7f6bb8a09700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6bb4067ec0 0x7f6bb406dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:53.733 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.731+0000 7f6bb920a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6bb40687d0 0x7f6bb4067630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38540/0 (socket says 172.21.15.27:38540) 2023-10-06T01:41:53.733 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.731+0000 7f6bb920a700 1 -- 172.21.15.27:0/4285652731 learned_addr learned my addr 172.21.15.27:0/4285652731 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:53.733 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.731+0000 7f6bb3fff700 1 --2- 172.21.15.27:0/4285652731 >> v2:172.21.15.124:3300/0 conn(0x7f6bb40c6b90 0x7f6bb4068290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:53.733 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.731+0000 7f6bb8a09700 1 -- 172.21.15.27:0/4285652731 >> v2:172.21.15.124:3300/0 conn(0x7f6bb40c6b90 msgr2=0x7f6bb4068290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:53.733 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.731+0000 7f6bb8a09700 1 --2- 172.21.15.27:0/4285652731 >> v2:172.21.15.124:3300/0 conn(0x7f6bb40c6b90 0x7f6bb4068290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:53.734 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.731+0000 7f6bb8a09700 1 -- 172.21.15.27:0/4285652731 >> v1:172.21.15.27:6789/0 conn(0x7f6bb40687d0 legacy=0x7f6bb4067630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:53.734 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.731+0000 7f6bb37fe700 1 -- 172.21.15.27:0/4285652731 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1303254726 0 0) 0x7f6bb4156000 con 0x7f6bb40687d0 2023-10-06T01:41:53.734 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.731+0000 7f6bb8a09700 1 -- 172.21.15.27:0/4285652731 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb406e5e0 con 0x7f6bb4067ec0 2023-10-06T01:41:53.734 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.731+0000 7f6bb3fff700 1 --2- 172.21.15.27:0/4285652731 >> v2:172.21.15.124:3300/0 conn(0x7f6bb40c6b90 0x7f6bb4068290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:41:53.734 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.731+0000 7f6bb8a09700 1 --2- 172.21.15.27:0/4285652731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bb4067ec0 0x7f6bb406dd70 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6bb4065980 tx=0x7f6bac009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=458bc3f29fad0988 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:53.734 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.731+0000 7f6bb37fe700 1 -- 172.21.15.27:0/4285652731 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bac00e070 con 0x7f6bb4067ec0 2023-10-06T01:41:53.734 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.731+0000 7f6bb37fe700 1 -- 172.21.15.27:0/4285652731 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bac00bbf0 con 0x7f6bb4067ec0 2023-10-06T01:41:53.734 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.731+0000 7f6bb37fe700 1 -- 172.21.15.27:0/4285652731 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bac012430 con 0x7f6bb4067ec0 2023-10-06T01:41:53.734 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.732+0000 7f6bbac6d700 1 -- 172.21.15.27:0/4285652731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bb4067ec0 msgr2=0x7f6bb406dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:53.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.732+0000 7f6bbac6d700 1 --2- 172.21.15.27:0/4285652731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bb4067ec0 0x7f6bb406dd70 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6bb4065980 tx=0x7f6bac009780 comp rx=0 tx=0).stop 2023-10-06T01:41:53.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.732+0000 7f6bbac6d700 1 -- 172.21.15.27:0/4285652731 shutdown_connections 2023-10-06T01:41:53.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.732+0000 7f6bbac6d700 1 --2- 172.21.15.27:0/4285652731 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bb4067ec0 0x7f6bb406dd70 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:53.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.732+0000 7f6bbac6d700 1 --2- 172.21.15.27:0/4285652731 >> v2:172.21.15.124:3300/0 conn(0x7f6bb40c6b90 0x7f6bb4068290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:53.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.732+0000 7f6bbac6d700 1 -- 172.21.15.27:0/4285652731 >> 172.21.15.27:0/4285652731 conn(0x7f6bb40c8d10 msgr2=0x7f6bb40c9110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:53.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.732+0000 7f6bbac6d700 1 -- 172.21.15.27:0/4285652731 shutdown_connections 2023-10-06T01:41:53.735 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.732+0000 7f6bbac6d700 1 -- 172.21.15.27:0/4285652731 wait complete. 2023-10-06T01:41:53.738 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.733+0000 7f6bbac6d700 1 Processor -- start 2023-10-06T01:41:53.738 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.733+0000 7f6bbac6d700 1 -- start start 2023-10-06T01:41:53.738 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.734+0000 7f6bbac6d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bb4067ec0 0x7f6bb40a80e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:53.738 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.734+0000 7f6bbac6d700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6bb40687d0 0x7f6bb4167de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:53.738 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.734+0000 7f6bbac6d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6bb40c6b90 0x7f6bb40ab4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:53.738 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.734+0000 7f6bbac6d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb40a8d30 con 0x7f6bb4067ec0 2023-10-06T01:41:53.738 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.734+0000 7f6bbac6d700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb41684f0 con 0x7f6bb40687d0 2023-10-06T01:41:53.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.734+0000 7f6bbac6d700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb4148880 con 0x7f6bb40c6b90 2023-10-06T01:41:53.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.734+0000 7f6bb8a09700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bb4067ec0 0x7f6bb40a80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:53.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.734+0000 7f6bb8a09700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bb4067ec0 0x7f6bb40a80e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39630/0 (socket says 172.21.15.27:39630) 2023-10-06T01:41:53.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.734+0000 7f6bb8a09700 1 -- 172.21.15.27:0/1194033548 learned_addr learned my addr 172.21.15.27:0/1194033548 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:53.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.734+0000 7f6bb3fff700 1 --2- 172.21.15.27:0/1194033548 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6bb40687d0 0x7f6bb4167de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:53.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.734+0000 7f6bb920a700 1 --2- 172.21.15.27:0/1194033548 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6bb40c6b90 0x7f6bb40ab4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:53.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.734+0000 7f6bb8a09700 1 -- 172.21.15.27:0/1194033548 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6bb40c6b90 msgr2=0x7f6bb40ab4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:53.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.734+0000 7f6bb8a09700 1 --2- 172.21.15.27:0/1194033548 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6bb40c6b90 0x7f6bb40ab4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:53.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.734+0000 7f6bb8a09700 1 -- 172.21.15.27:0/1194033548 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6bb40687d0 msgr2=0x7f6bb4167de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:53.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.734+0000 7f6bb8a09700 1 --2- 172.21.15.27:0/1194033548 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6bb40687d0 0x7f6bb4167de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:53.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.734+0000 7f6bb8a09700 1 -- 172.21.15.27:0/1194033548 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bac009a20 con 0x7f6bb4067ec0 2023-10-06T01:41:53.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.734+0000 7f6bb8a09700 1 --2- 172.21.15.27:0/1194033548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bb4067ec0 0x7f6bb40a80e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6bb406d1a0 tx=0x7f6bac019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:53.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.735+0000 7f6bb0ff9700 1 -- 172.21.15.27:0/1194033548 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bac01b420 con 0x7f6bb4067ec0 2023-10-06T01:41:53.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.735+0000 7f6bb0ff9700 1 -- 172.21.15.27:0/1194033548 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bac01b580 con 0x7f6bb4067ec0 2023-10-06T01:41:53.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.735+0000 7f6bbac6d700 1 -- 172.21.15.27:0/1194033548 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bb40acf60 con 0x7f6bb4067ec0 2023-10-06T01:41:53.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.735+0000 7f6bb0ff9700 1 -- 172.21.15.27:0/1194033548 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bac01b940 con 0x7f6bb4067ec0 2023-10-06T01:41:53.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.735+0000 7f6bbac6d700 1 -- 172.21.15.27:0/1194033548 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bb40a8a50 con 0x7f6bb4067ec0 2023-10-06T01:41:53.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.736+0000 7f6bbac6d700 1 -- 172.21.15.27:0/1194033548 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bb40717c0 con 0x7f6bb4067ec0 2023-10-06T01:41:53.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.736+0000 7f6bb0ff9700 1 -- 172.21.15.27:0/1194033548 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6bac01bcb0 con 0x7f6bb4067ec0 2023-10-06T01:41:53.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.736+0000 7f6bb0ff9700 1 --2- 172.21.15.27:0/1194033548 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6b94066f30 0x7f6b940693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:53.740 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.737+0000 7f6bb0ff9700 1 -- 172.21.15.27:0/1194033548 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6144+0+0 (secure 0 0 0) 0x7f6bac0864b0 con 0x7f6bb4067ec0 2023-10-06T01:41:53.741 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.737+0000 7f6bb3fff700 1 --2- 172.21.15.27:0/1194033548 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6b94066f30 0x7f6b940693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:53.741 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.737+0000 7f6bb3fff700 1 --2- 172.21.15.27:0/1194033548 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6b94066f30 0x7f6b940693e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6bb40b1800 tx=0x7f6ba4006f90 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:53.741 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.741+0000 7f6bb0ff9700 1 -- 172.21.15.27:0/1194033548 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6bac0534f0 con 0x7f6bb4067ec0 2023-10-06T01:41:53.934 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:53.934+0000 7f6bbac6d700 1 -- 172.21.15.27:0/1194033548 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6bb4067b00 con 0x7f6bb4067ec0 2023-10-06T01:41:54.594 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.593+0000 7f6bb0ff9700 1 -- 172.21.15.27:0/1194033548 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v22) v1 ==== 181+0+0 (secure 0 0 0) 0x7f6bac009880 con 0x7f6bb4067ec0 2023-10-06T01:41:54.598 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.597+0000 7f6bbac6d700 1 -- 172.21.15.27:0/1194033548 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6b94066f30 msgr2=0x7f6b940693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:54.598 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.597+0000 7f6bbac6d700 1 --2- 172.21.15.27:0/1194033548 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6b94066f30 0x7f6b940693e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6bb40b1800 tx=0x7f6ba4006f90 comp rx=0 tx=0).stop 2023-10-06T01:41:54.598 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.597+0000 7f6bbac6d700 1 -- 172.21.15.27:0/1194033548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bb4067ec0 msgr2=0x7f6bb40a80e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:54.598 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.597+0000 7f6bbac6d700 1 --2- 172.21.15.27:0/1194033548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bb4067ec0 0x7f6bb40a80e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6bb406d1a0 tx=0x7f6bac019000 comp rx=0 tx=0).stop 2023-10-06T01:41:54.598 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.597+0000 7f6bbac6d700 1 -- 172.21.15.27:0/1194033548 shutdown_connections 2023-10-06T01:41:54.598 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.597+0000 7f6bbac6d700 1 --2- 172.21.15.27:0/1194033548 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6b94066f30 0x7f6b940693e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:54.598 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.597+0000 7f6bbac6d700 1 --2- 172.21.15.27:0/1194033548 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6bb40c6b90 0x7f6bb40ab4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:54.599 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.597+0000 7f6bbac6d700 1 --2- 172.21.15.27:0/1194033548 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bb4067ec0 0x7f6bb40a80e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:54.599 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.597+0000 7f6bbac6d700 1 --2- 172.21.15.27:0/1194033548 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6bb40687d0 0x7f6bb4167de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:54.599 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.597+0000 7f6bbac6d700 1 -- 172.21.15.27:0/1194033548 >> 172.21.15.27:0/1194033548 conn(0x7f6bb40c8d10 msgr2=0x7f6bb40c90e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:54.599 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.598+0000 7f6bbac6d700 1 -- 172.21.15.27:0/1194033548 shutdown_connections 2023-10-06T01:41:54.599 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.598+0000 7f6bbac6d700 1 -- 172.21.15.27:0/1194033548 wait complete. 2023-10-06T01:41:54.599 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' removed 2023-10-06T01:41:54.613 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-06T01:41:54.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.738+0000 7f075cefc700 1 Processor -- start 2023-10-06T01:41:54.739 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.738+0000 7f075cefc700 1 -- start start 2023-10-06T01:41:54.745 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.739+0000 7f075cefc700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f07580b55b0 0x7f07580ccbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:54.745 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.739+0000 7f075cefc700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f07580cf800 0x7f07580b5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:54.745 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.739+0000 7f075cefc700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07580cbed0 con 0x7f07580d0f00 2023-10-06T01:41:54.746 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.739+0000 7f075cefc700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0758159d40 con 0x7f07580b55b0 2023-10-06T01:41:54.746 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.739+0000 7f075cefc700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07580cc750 con 0x7f07580cf800 2023-10-06T01:41:54.746 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.740+0000 7f0756d9d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f07580d0f00 0x7f07580ce270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:38550/0 (socket says 172.21.15.27:38550) 2023-10-06T01:41:54.746 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.740+0000 7f0756d9d700 1 -- 172.21.15.27:0/1337239458 learned_addr learned my addr 172.21.15.27:0/1337239458 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:54.746 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.740+0000 7f075559a700 1 -- 172.21.15.27:0/1337239458 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2029916158 0 0) 0x7f07580cbed0 con 0x7f07580d0f00 2023-10-06T01:41:54.746 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.740+0000 7f075559a700 1 -- 172.21.15.27:0/1337239458 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0744003570 con 0x7f07580d0f00 2023-10-06T01:41:54.746 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.740+0000 7f075659c700 1 --2- 172.21.15.27:0/1337239458 >> v2:172.21.15.124:3300/0 conn(0x7f07580b55b0 0x7f07580ccbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:54.746 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.740+0000 7f0755d9b700 1 --2- 172.21.15.27:0/1337239458 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f07580cf800 0x7f07580b5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:54.746 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.740+0000 7f075559a700 1 -- 172.21.15.27:0/1337239458 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1608563863 0 0) 0x7f0744003570 con 0x7f07580d0f00 2023-10-06T01:41:54.747 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.740+0000 7f075559a700 1 -- 172.21.15.27:0/1337239458 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f07580cf800 msgr2=0x7f07580b5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:54.747 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.740+0000 7f075559a700 1 --2- 172.21.15.27:0/1337239458 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f07580cf800 0x7f07580b5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:54.747 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.740+0000 7f075559a700 1 -- 172.21.15.27:0/1337239458 >> v2:172.21.15.124:3300/0 conn(0x7f07580b55b0 msgr2=0x7f07580ccbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:54.747 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.740+0000 7f075559a700 1 --2- 172.21.15.27:0/1337239458 >> v2:172.21.15.124:3300/0 conn(0x7f07580b55b0 0x7f07580ccbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:54.747 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.740+0000 7f075559a700 1 -- 172.21.15.27:0/1337239458 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07580b5ec0 con 0x7f07580d0f00 2023-10-06T01:41:54.747 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.740+0000 7f075559a700 1 -- 172.21.15.27:0/1337239458 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2619078944 0 0) 0x7f07480017f0 con 0x7f07580d0f00 2023-10-06T01:41:54.747 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.740+0000 7f075559a700 1 -- 172.21.15.27:0/1337239458 >> v1:172.21.15.27:6789/0 conn(0x7f07580d0f00 legacy=0x7f07580ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:54.747 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.740+0000 7f075559a700 1 --2- 172.21.15.27:0/1337239458 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f07580cf800 0x7f0744003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:54.748 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.740+0000 7f075559a700 1 --2- 172.21.15.27:0/1337239458 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0744003fe0 0x7f0744004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:54.748 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.740+0000 7f075559a700 1 --2- 172.21.15.27:0/1337239458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0744004d50 0x7f07440051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:54.748 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.740+0000 7f075559a700 1 -- 172.21.15.27:0/1337239458 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07580cc750 con 0x7f0744004d50 2023-10-06T01:41:54.748 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.740+0000 7f075559a700 1 -- 172.21.15.27:0/1337239458 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f074400bb70 con 0x7f0744003fe0 2023-10-06T01:41:54.748 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.740+0000 7f075559a700 1 -- 172.21.15.27:0/1337239458 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f074400bd30 con 0x7f07580cf800 2023-10-06T01:41:54.748 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.740+0000 7f0756d9d700 1 --2- 172.21.15.27:0/1337239458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0744004d50 0x7f07440051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:54.748 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.740+0000 7f075659c700 1 --2- 172.21.15.27:0/1337239458 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f07580cf800 0x7f0744003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:54.748 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.740+0000 7f0755d9b700 1 --2- 172.21.15.27:0/1337239458 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0744003fe0 0x7f0744004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:54.749 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.741+0000 7f0756d9d700 1 -- 172.21.15.27:0/1337239458 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f07580cf800 msgr2=0x7f0744003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:54.749 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.741+0000 7f0756d9d700 1 --2- 172.21.15.27:0/1337239458 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f07580cf800 0x7f0744003a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:54.749 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.741+0000 7f0756d9d700 1 -- 172.21.15.27:0/1337239458 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0744003fe0 msgr2=0x7f0744004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:54.749 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.741+0000 7f0756d9d700 1 --2- 172.21.15.27:0/1337239458 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0744003fe0 0x7f0744004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:54.749 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.741+0000 7f0756d9d700 1 -- 172.21.15.27:0/1337239458 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0744003570 con 0x7f0744004d50 2023-10-06T01:41:54.749 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.741+0000 7f0756d9d700 1 --2- 172.21.15.27:0/1337239458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0744004d50 0x7f07440051c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f074800a940 tx=0x7f0748002210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb9dcabc9a5dac7 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:54.749 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.741+0000 7f075559a700 1 -- 172.21.15.27:0/1337239458 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0748007110 con 0x7f0744004d50 2023-10-06T01:41:54.749 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.741+0000 7f075559a700 1 -- 172.21.15.27:0/1337239458 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0748007290 con 0x7f0744004d50 2023-10-06T01:41:54.750 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.741+0000 7f075559a700 1 -- 172.21.15.27:0/1337239458 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0748007650 con 0x7f0744004d50 2023-10-06T01:41:54.750 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.741+0000 7f075cefc700 1 -- 172.21.15.27:0/1337239458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0744004d50 msgr2=0x7f07440051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:54.750 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.741+0000 7f075cefc700 1 --2- 172.21.15.27:0/1337239458 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0744004d50 0x7f07440051c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f074800a940 tx=0x7f0748002210 comp rx=0 tx=0).stop 2023-10-06T01:41:54.750 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.741+0000 7f0756d9d700 1 -- 172.21.15.27:0/1337239458 reap_dead start 2023-10-06T01:41:54.750 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.741+0000 7f075cefc700 1 -- 172.21.15.27:0/1337239458 shutdown_connections 2023-10-06T01:41:54.750 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.742+0000 7f075cefc700 1 -- 172.21.15.27:0/1337239458 >> 172.21.15.27:0/1337239458 conn(0x7f075808a050 msgr2=0x7f0758099970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:54.750 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.742+0000 7f075cefc700 1 -- 172.21.15.27:0/1337239458 shutdown_connections 2023-10-06T01:41:54.750 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.742+0000 7f075cefc700 1 -- 172.21.15.27:0/1337239458 wait complete. 2023-10-06T01:41:54.750 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.743+0000 7f075cefc700 1 Processor -- start 2023-10-06T01:41:54.750 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.743+0000 7f075cefc700 1 -- start start 2023-10-06T01:41:54.751 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.743+0000 7f075cefc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0744004d50 0x7f075814c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:54.751 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.743+0000 7f075cefc700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f075806a280 0x7f07580afdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:54.751 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.743+0000 7f075cefc700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f075815aae0 0x7f075815af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:54.751 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.743+0000 7f075cefc700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075814c730 con 0x7f0744004d50 2023-10-06T01:41:54.751 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.743+0000 7f075cefc700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075815b4a0 con 0x7f075815aae0 2023-10-06T01:41:54.751 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.743+0000 7f075cefc700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07580b1880 con 0x7f075806a280 2023-10-06T01:41:54.751 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.743+0000 7f075659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0744004d50 0x7f075814c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:54.751 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.744+0000 7f075659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0744004d50 0x7f075814c1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:39648/0 (socket says 172.21.15.27:39648) 2023-10-06T01:41:54.752 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.744+0000 7f075659c700 1 -- 172.21.15.27:0/3776127274 learned_addr learned my addr 172.21.15.27:0/3776127274 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:54.752 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.744+0000 7f0756d9d700 1 --2- 172.21.15.27:0/3776127274 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f075815aae0 0x7f075815af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:54.752 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.744+0000 7f0755d9b700 1 --2- 172.21.15.27:0/3776127274 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f075806a280 0x7f07580afdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:54.752 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.744+0000 7f075659c700 1 -- 172.21.15.27:0/3776127274 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f075806a280 msgr2=0x7f07580afdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:54.752 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.744+0000 7f075659c700 1 --2- 172.21.15.27:0/3776127274 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f075806a280 0x7f07580afdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:54.752 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.744+0000 7f075659c700 1 -- 172.21.15.27:0/3776127274 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f075815aae0 msgr2=0x7f075815af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:54.752 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.744+0000 7f075659c700 1 --2- 172.21.15.27:0/3776127274 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f075815aae0 0x7f075815af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:54.752 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.744+0000 7f075659c700 1 -- 172.21.15.27:0/3776127274 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07580b1b20 con 0x7f0744004d50 2023-10-06T01:41:54.752 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.744+0000 7f075659c700 1 --2- 172.21.15.27:0/3776127274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0744004d50 0x7f075814c1f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0750011a40 tx=0x7f0750009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:54.752 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.744+0000 7f074e7fc700 1 -- 172.21.15.27:0/3776127274 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0750011e80 con 0x7f0744004d50 2023-10-06T01:41:54.753 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.744+0000 7f074e7fc700 1 -- 172.21.15.27:0/3776127274 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0750005c10 con 0x7f0744004d50 2023-10-06T01:41:54.753 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.744+0000 7f075cefc700 1 -- 172.21.15.27:0/3776127274 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07580b1e00 con 0x7f0744004d50 2023-10-06T01:41:54.753 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.744+0000 7f074e7fc700 1 -- 172.21.15.27:0/3776127274 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075000f6d0 con 0x7f0744004d50 2023-10-06T01:41:54.753 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.744+0000 7f075cefc700 1 -- 172.21.15.27:0/3776127274 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07580b23c0 con 0x7f0744004d50 2023-10-06T01:41:54.753 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.745+0000 7f075cefc700 1 -- 172.21.15.27:0/3776127274 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07580717c0 con 0x7f0744004d50 2023-10-06T01:41:54.754 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.746+0000 7f074e7fc700 1 -- 172.21.15.27:0/3776127274 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f075000fb20 con 0x7f0744004d50 2023-10-06T01:41:54.754 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.746+0000 7f074e7fc700 1 --2- 172.21.15.27:0/3776127274 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0738066f30 0x7f07380693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:54.754 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.746+0000 7f074e7fc700 1 -- 172.21.15.27:0/3776127274 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5706+0+0 (secure 0 0 0) 0x7f0750084df0 con 0x7f0744004d50 2023-10-06T01:41:54.754 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.746+0000 7f0755d9b700 1 --2- 172.21.15.27:0/3776127274 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0738066f30 0x7f07380693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:54.754 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.747+0000 7f0755d9b700 1 --2- 172.21.15.27:0/3776127274 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0738066f30 0x7f07380693e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f075806a650 tx=0x7f0740008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:54.755 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.750+0000 7f074e7fc700 1 -- 172.21.15.27:0/3776127274 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0750052060 con 0x7f0744004d50 2023-10-06T01:41:54.944 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:54.944+0000 7f075cefc700 1 -- 172.21.15.27:0/3776127274 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f07580cffe0 con 0x7f0744004d50 2023-10-06T01:41:55.598 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.598+0000 7f074e7fc700 1 -- 172.21.15.27:0/3776127274 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v23) v1 ==== 169+0+0 (secure 0 0 0) 0x7f07500210d0 con 0x7f0744004d50 2023-10-06T01:41:55.607 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.602+0000 7f075cefc700 1 -- 172.21.15.27:0/3776127274 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0738066f30 msgr2=0x7f07380693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:55.607 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.602+0000 7f075cefc700 1 --2- 172.21.15.27:0/3776127274 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0738066f30 0x7f07380693e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f075806a650 tx=0x7f0740008040 comp rx=0 tx=0).stop 2023-10-06T01:41:55.607 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.602+0000 7f075cefc700 1 -- 172.21.15.27:0/3776127274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0744004d50 msgr2=0x7f075814c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:55.607 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.602+0000 7f075cefc700 1 --2- 172.21.15.27:0/3776127274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0744004d50 0x7f075814c1f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0750011a40 tx=0x7f0750009120 comp rx=0 tx=0).stop 2023-10-06T01:41:55.607 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.602+0000 7f075cefc700 1 -- 172.21.15.27:0/3776127274 shutdown_connections 2023-10-06T01:41:55.608 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.602+0000 7f075cefc700 1 --2- 172.21.15.27:0/3776127274 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0738066f30 0x7f07380693e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:55.608 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.602+0000 7f075cefc700 1 --2- 172.21.15.27:0/3776127274 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f075815aae0 0x7f075815af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:55.608 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.602+0000 7f075cefc700 1 --2- 172.21.15.27:0/3776127274 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0744004d50 0x7f075814c1f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:55.608 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.602+0000 7f075cefc700 1 --2- 172.21.15.27:0/3776127274 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f075806a280 0x7f07580afdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:55.608 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.602+0000 7f075cefc700 1 -- 172.21.15.27:0/3776127274 >> 172.21.15.27:0/3776127274 conn(0x7f075808a050 msgr2=0x7f0758158ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:55.608 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.602+0000 7f075cefc700 1 -- 172.21.15.27:0/3776127274 shutdown_connections 2023-10-06T01:41:55.608 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.602+0000 7f075cefc700 1 -- 172.21.15.27:0/3776127274 wait complete. 2023-10-06T01:41:55.608 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' removed 2023-10-06T01:41:55.617 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-06T01:41:55.617 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-06T01:41:55.617 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-06T01:41:55.617 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T01:41:55.618 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-06T01:41:55.618 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-06T01:41:55.618 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-06T01:41:55.618 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T01:41:55.618 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-06T01:41:55.618 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-06T01:41:55.618 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-06T01:41:55.619 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T01:41:55.619 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-06T01:41:55.619 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-06T01:41:55.619 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-06T01:41:55.619 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T01:41:55.627 INFO:tasks.ceph.mds.a:Stopped 2023-10-06T01:41:55.627 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-06T01:41:55.628 INFO:tasks.ceph.mds.c:Stopped 2023-10-06T01:41:55.628 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-06T01:41:55.632 INFO:tasks.ceph.mds.b:Stopped 2023-10-06T01:41:55.633 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-06T01:41:55.633 INFO:tasks.ceph.mds.d:Stopped 2023-10-06T01:41:55.633 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-06T01:41:55.669 INFO:tasks.ceph.mds.a:Started 2023-10-06T01:41:55.670 DEBUG:teuthology.parallel:result is None 2023-10-06T01:41:55.670 INFO:tasks.ceph.mds.c:Started 2023-10-06T01:41:55.670 DEBUG:teuthology.parallel:result is None 2023-10-06T01:41:55.674 INFO:tasks.ceph.mds.b:Started 2023-10-06T01:41:55.674 DEBUG:teuthology.parallel:result is None 2023-10-06T01:41:55.675 INFO:tasks.ceph.mds.d:Started 2023-10-06T01:41:55.675 DEBUG:teuthology.parallel:result is None 2023-10-06T01:41:55.676 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-06T01:41:55.746 INFO:tasks.ceph.mds.c.smithi027.stdout:starting mds.c at 2023-10-06T01:41:55.746 INFO:tasks.ceph.mds.a.smithi027.stdout:starting mds.a at 2023-10-06T01:41:55.761 INFO:tasks.ceph.mds.b.smithi124.stdout:starting mds.b at 2023-10-06T01:41:55.761 INFO:tasks.ceph.mds.d.smithi124.stdout:starting mds.d at 2023-10-06T01:41:55.800 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.800+0000 7f449b59e700 1 Processor -- start 2023-10-06T01:41:55.800 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.800+0000 7f449b59e700 1 -- start start 2023-10-06T01:41:55.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.801+0000 7f449b59e700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f449c154110 0x7f449c0c9b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:55.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.801+0000 7f449b59e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f449c0ca050 0x7f449c067b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:55.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.801+0000 7f449b59e700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f449c15e310 con 0x7f449c153d40 2023-10-06T01:41:55.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.801+0000 7f449b59e700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f449c0ca4d0 con 0x7f449c154110 2023-10-06T01:41:55.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.801+0000 7f449b59e700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f449c15d4b0 con 0x7f449c0ca050 2023-10-06T01:41:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.801+0000 7f449a59c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f449c153d40 0x7f449c0aa110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54392/0 (socket says 172.21.15.27:54392) 2023-10-06T01:41:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.801+0000 7f449a59c700 1 -- 172.21.15.27:0/2537661765 learned_addr learned my addr 172.21.15.27:0/2537661765 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.801+0000 7f449ad9d700 1 --2- 172.21.15.27:0/2537661765 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f449c0ca050 0x7f449c067b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.801+0000 7f4499d9b700 1 --2- 172.21.15.27:0/2537661765 >> v2:172.21.15.124:3300/0 conn(0x7f449c154110 0x7f449c0c9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.801+0000 7f449959a700 1 -- 172.21.15.27:0/2537661765 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4139456940 0 0) 0x7f449c15e310 con 0x7f449c153d40 2023-10-06T01:41:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.801+0000 7f449959a700 1 -- 172.21.15.27:0/2537661765 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4488003570 con 0x7f449c153d40 2023-10-06T01:41:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.801+0000 7f449ad9d700 1 -- 172.21.15.27:0/2537661765 >> v2:172.21.15.124:3300/0 conn(0x7f449c154110 msgr2=0x7f449c0c9b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.801+0000 7f449ad9d700 1 --2- 172.21.15.27:0/2537661765 >> v2:172.21.15.124:3300/0 conn(0x7f449c154110 0x7f449c0c9b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.801+0000 7f449ad9d700 1 -- 172.21.15.27:0/2537661765 >> v1:172.21.15.27:6789/0 conn(0x7f449c153d40 legacy=0x7f449c0aa110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.801+0000 7f449ad9d700 1 -- 172.21.15.27:0/2537661765 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f449c15d700 con 0x7f449c0ca050 2023-10-06T01:41:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.801+0000 7f4499d9b700 1 --2- 172.21.15.27:0/2537661765 >> v2:172.21.15.124:3300/0 conn(0x7f449c154110 0x7f449c0c9b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:41:55.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.802+0000 7f449ad9d700 1 --2- 172.21.15.27:0/2537661765 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f449c0ca050 0x7f449c067b00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f448c009e60 tx=0x7f448c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a6f8f94daba757e server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:55.804 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.802+0000 7f449959a700 1 -- 172.21.15.27:0/2537661765 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f448c00e070 con 0x7f449c0ca050 2023-10-06T01:41:55.804 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.802+0000 7f449959a700 1 -- 172.21.15.27:0/2537661765 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f448c00bab0 con 0x7f449c0ca050 2023-10-06T01:41:55.804 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.802+0000 7f449959a700 1 -- 172.21.15.27:0/2537661765 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f448c00be70 con 0x7f449c0ca050 2023-10-06T01:41:55.806 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.802+0000 7f449b59e700 1 -- 172.21.15.27:0/2537661765 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f449c0ca050 msgr2=0x7f449c067b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:55.810 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.802+0000 7f449b59e700 1 --2- 172.21.15.27:0/2537661765 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f449c0ca050 0x7f449c067b00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f448c009e60 tx=0x7f448c009780 comp rx=0 tx=0).stop 2023-10-06T01:41:55.810 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.802+0000 7f449b59e700 1 -- 172.21.15.27:0/2537661765 shutdown_connections 2023-10-06T01:41:55.810 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.802+0000 7f449b59e700 1 --2- 172.21.15.27:0/2537661765 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f449c0ca050 0x7f449c067b00 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:55.810 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.802+0000 7f449b59e700 1 --2- 172.21.15.27:0/2537661765 >> v2:172.21.15.124:3300/0 conn(0x7f449c154110 0x7f449c0c9b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:55.810 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.802+0000 7f449b59e700 1 -- 172.21.15.27:0/2537661765 >> 172.21.15.27:0/2537661765 conn(0x7f449c06a970 msgr2=0x7f449c0c9400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:55.811 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.803+0000 7f449b59e700 1 -- 172.21.15.27:0/2537661765 shutdown_connections 2023-10-06T01:41:55.811 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.803+0000 7f449b59e700 1 -- 172.21.15.27:0/2537661765 wait complete. 2023-10-06T01:41:55.811 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.804+0000 7f449b59e700 1 Processor -- start 2023-10-06T01:41:55.811 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.804+0000 7f449b59e700 1 -- start start 2023-10-06T01:41:55.811 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.804+0000 7f449b59e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f449c0ca050 0x7f449c140ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:55.811 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.804+0000 7f449b59e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f449c153d40 0x7f449c150fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:55.811 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.804+0000 7f449b59e700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f449c154110 0x7f449c15c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:55.811 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.804+0000 7f449b59e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449c0aca70 con 0x7f449c0ca050 2023-10-06T01:41:55.811 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.804+0000 7f449b59e700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449c0abdf0 con 0x7f449c154110 2023-10-06T01:41:55.811 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.804+0000 7f449b59e700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f449c0ac0d0 con 0x7f449c153d40 2023-10-06T01:41:55.811 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.805+0000 7f449a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f449c0ca050 0x7f449c140ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:55.812 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.805+0000 7f449a59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f449c0ca050 0x7f449c140ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53822/0 (socket says 172.21.15.27:53822) 2023-10-06T01:41:55.812 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.805+0000 7f449a59c700 1 -- 172.21.15.27:0/4209078479 learned_addr learned my addr 172.21.15.27:0/4209078479 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:55.812 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.805+0000 7f4499d9b700 1 --2- 172.21.15.27:0/4209078479 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f449c153d40 0x7f449c150fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:55.812 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.805+0000 7f449ad9d700 1 --2- 172.21.15.27:0/4209078479 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f449c154110 0x7f449c15c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:55.812 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.805+0000 7f449a59c700 1 -- 172.21.15.27:0/4209078479 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f449c153d40 msgr2=0x7f449c150fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:55.812 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.805+0000 7f449a59c700 1 --2- 172.21.15.27:0/4209078479 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f449c153d40 0x7f449c150fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:55.812 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.805+0000 7f449a59c700 1 -- 172.21.15.27:0/4209078479 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f449c154110 msgr2=0x7f449c15c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:55.812 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.805+0000 7f449a59c700 1 --2- 172.21.15.27:0/4209078479 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f449c154110 0x7f449c15c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:55.812 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.805+0000 7f449a59c700 1 -- 172.21.15.27:0/4209078479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4484009420 con 0x7f449c0ca050 2023-10-06T01:41:55.812 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.805+0000 7f449a59c700 1 --2- 172.21.15.27:0/4209078479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f449c0ca050 0x7f449c140ef0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f449400c7c0 tx=0x7f4494007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:55.813 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.805+0000 7f44927fc700 1 -- 172.21.15.27:0/4209078479 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4494005cd0 con 0x7f449c0ca050 2023-10-06T01:41:55.813 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.805+0000 7f44927fc700 1 -- 172.21.15.27:0/4209078479 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4494005e30 con 0x7f449c0ca050 2023-10-06T01:41:55.813 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.805+0000 7f449b59e700 1 -- 172.21.15.27:0/4209078479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f448c009a20 con 0x7f449c0ca050 2023-10-06T01:41:55.813 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.805+0000 7f44927fc700 1 -- 172.21.15.27:0/4209078479 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4494007320 con 0x7f449c0ca050 2023-10-06T01:41:55.813 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.805+0000 7f449b59e700 1 -- 172.21.15.27:0/4209078479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f449c0ac690 con 0x7f449c0ca050 2023-10-06T01:41:55.813 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.806+0000 7f449b59e700 1 -- 172.21.15.27:0/4209078479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f449c0acbb0 con 0x7f449c0ca050 2023-10-06T01:41:55.814 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.807+0000 7f44927fc700 1 -- 172.21.15.27:0/4209078479 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f449401b3b0 con 0x7f449c0ca050 2023-10-06T01:41:55.814 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.807+0000 7f44927fc700 1 --2- 172.21.15.27:0/4209078479 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f447c066f30 0x7f447c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:55.814 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.807+0000 7f44927fc700 1 -- 172.21.15.27:0/4209078479 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5292+0+0 (secure 0 0 0) 0x7f4494083a30 con 0x7f449c0ca050 2023-10-06T01:41:55.814 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.807+0000 7f4499d9b700 1 --2- 172.21.15.27:0/4209078479 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f447c066f30 0x7f447c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:55.814 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.808+0000 7f4499d9b700 1 --2- 172.21.15.27:0/4209078479 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f447c066f30 0x7f447c0693e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f449c0cb2b0 tx=0x7f44840091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:55.814 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:55.811+0000 7f44927fc700 1 -- 172.21.15.27:0/4209078479 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4494050e40 con 0x7f449c0ca050 2023-10-06T01:41:56.032 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.031+0000 7f449b59e700 1 -- 172.21.15.27:0/4209078479 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f449c04c3c0 con 0x7f449c0ca050 2023-10-06T01:41:56.606 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.606+0000 7f44927fc700 1 -- 172.21.15.27:0/4209078479 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v25) v1 ==== 97+0+0 (secure 0 0 0) 0x7f4494017020 con 0x7f449c0ca050 2023-10-06T01:41:56.609 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.609+0000 7f449b59e700 1 -- 172.21.15.27:0/4209078479 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f447c066f30 msgr2=0x7f447c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:56.609 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.609+0000 7f449b59e700 1 --2- 172.21.15.27:0/4209078479 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f447c066f30 0x7f447c0693e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f449c0cb2b0 tx=0x7f44840091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:56.610 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.609+0000 7f449b59e700 1 -- 172.21.15.27:0/4209078479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f449c0ca050 msgr2=0x7f449c140ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:56.610 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.609+0000 7f449b59e700 1 --2- 172.21.15.27:0/4209078479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f449c0ca050 0x7f449c140ef0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f449400c7c0 tx=0x7f4494007e00 comp rx=0 tx=0).stop 2023-10-06T01:41:56.610 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.609+0000 7f449b59e700 1 -- 172.21.15.27:0/4209078479 shutdown_connections 2023-10-06T01:41:56.610 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.609+0000 7f449b59e700 1 --2- 172.21.15.27:0/4209078479 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f447c066f30 0x7f447c0693e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:56.610 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.609+0000 7f449b59e700 1 --2- 172.21.15.27:0/4209078479 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f449c154110 0x7f449c15c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:56.610 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.609+0000 7f449b59e700 1 --2- 172.21.15.27:0/4209078479 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f449c0ca050 0x7f449c140ef0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:56.610 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.609+0000 7f449b59e700 1 --2- 172.21.15.27:0/4209078479 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f449c153d40 0x7f449c150fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:56.610 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.609+0000 7f449b59e700 1 -- 172.21.15.27:0/4209078479 >> 172.21.15.27:0/4209078479 conn(0x7f449c06a970 msgr2=0x7f449c06b300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:56.610 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.609+0000 7f449b59e700 1 -- 172.21.15.27:0/4209078479 shutdown_connections 2023-10-06T01:41:56.610 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.609+0000 7f449b59e700 1 -- 172.21.15.27:0/4209078479 wait complete. 2023-10-06T01:41:56.611 INFO:teuthology.orchestra.run.smithi027.stderr: removed all blocklist entries 2023-10-06T01:41:56.622 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-06T01:41:56.743 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.743+0000 7f256d885700 1 Processor -- start 2023-10-06T01:41:56.744 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.743+0000 7f256d885700 1 -- start start 2023-10-06T01:41:56.746 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.744+0000 7f256d885700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f256815d560 0x7f25680d0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:56.746 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.744+0000 7f256d885700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f25681591b0 0x7f2568159630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:56.747 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.744+0000 7f256d885700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25680b2ad0 con 0x7f256815d930 2023-10-06T01:41:56.747 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.744+0000 7f256d885700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25680b4160 con 0x7f256815d560 2023-10-06T01:41:56.747 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.744+0000 7f256d885700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2568159b70 con 0x7f25681591b0 2023-10-06T01:41:56.747 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.744+0000 7f25677fe700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f25681591b0 0x7f2568159630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:56.747 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.744+0000 7f2566ffd700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f256815d560 0x7f25680d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:56.747 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.744+0000 7f25667fc700 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f256815d930 0x7f25680b1a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.27:48816/0 (socket says 172.21.15.27:48816) 2023-10-06T01:41:56.747 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.744+0000 7f25667fc700 1 -- 172.21.15.27:0/3241195106 learned_addr learned my addr 172.21.15.27:0/3241195106 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:56.748 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.744+0000 7f25677fe700 1 -- 172.21.15.27:0/3241195106 >> v2:172.21.15.124:3300/0 conn(0x7f256815d560 msgr2=0x7f25680d0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:56.748 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.744+0000 7f25677fe700 1 --2- 172.21.15.27:0/3241195106 >> v2:172.21.15.124:3300/0 conn(0x7f256815d560 0x7f25680d0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:56.748 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.744+0000 7f25677fe700 1 -- 172.21.15.27:0/3241195106 >> v1:172.21.15.124:6789/0 conn(0x7f256815d930 legacy=0x7f25680b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:56.748 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.744+0000 7f25677fe700 1 -- 172.21.15.27:0/3241195106 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25680b2230 con 0x7f25681591b0 2023-10-06T01:41:56.748 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.744+0000 7f2566ffd700 1 --2- 172.21.15.27:0/3241195106 >> v2:172.21.15.124:3300/0 conn(0x7f256815d560 0x7f25680d0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:41:56.748 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.744+0000 7f2565ffb700 1 -- 172.21.15.27:0/3241195106 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2195654452 0 0) 0x7f25680b2ad0 con 0x7f256815d930 2023-10-06T01:41:56.748 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.745+0000 7f25677fe700 1 --2- 172.21.15.27:0/3241195106 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f25681591b0 0x7f2568159630 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f25580098e0 tx=0x7f25580091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cbc52ceacd03bf7 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:56.748 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.745+0000 7f2565ffb700 1 -- 172.21.15.27:0/3241195106 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255800b640 con 0x7f25681591b0 2023-10-06T01:41:56.748 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.745+0000 7f2565ffb700 1 -- 172.21.15.27:0/3241195106 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f255800b7a0 con 0x7f25681591b0 2023-10-06T01:41:56.748 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.745+0000 7f2565ffb700 1 -- 172.21.15.27:0/3241195106 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255800bb40 con 0x7f25681591b0 2023-10-06T01:41:56.749 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.745+0000 7f256d885700 1 -- 172.21.15.27:0/3241195106 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f25681591b0 msgr2=0x7f2568159630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:56.749 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.745+0000 7f256d885700 1 --2- 172.21.15.27:0/3241195106 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f25681591b0 0x7f2568159630 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f25580098e0 tx=0x7f25580091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:56.749 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.745+0000 7f256d885700 1 -- 172.21.15.27:0/3241195106 shutdown_connections 2023-10-06T01:41:56.749 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.745+0000 7f256d885700 1 --2- 172.21.15.27:0/3241195106 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f25681591b0 0x7f2568159630 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:56.749 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.745+0000 7f256d885700 1 --2- 172.21.15.27:0/3241195106 >> v2:172.21.15.124:3300/0 conn(0x7f256815d560 0x7f25680d0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:56.749 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.745+0000 7f256d885700 1 -- 172.21.15.27:0/3241195106 >> 172.21.15.27:0/3241195106 conn(0x7f25680d1180 msgr2=0x7f2568069640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:56.749 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.745+0000 7f256d885700 1 -- 172.21.15.27:0/3241195106 shutdown_connections 2023-10-06T01:41:56.749 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.745+0000 7f256d885700 1 -- 172.21.15.27:0/3241195106 wait complete. 2023-10-06T01:41:56.752 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.747+0000 7f256d885700 1 Processor -- start 2023-10-06T01:41:56.752 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.747+0000 7f256d885700 1 -- start start 2023-10-06T01:41:56.752 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.747+0000 7f256d885700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f25681591b0 0x7f2568149160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:56.752 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.747+0000 7f256d885700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f256815d560 0x7f25680a9af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:56.752 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.747+0000 7f256d885700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f256815d930 0x7f25680aa030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:56.752 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.747+0000 7f256d885700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25681698b0 con 0x7f256815d560 2023-10-06T01:41:56.753 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.747+0000 7f256d885700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25680a7c10 con 0x7f25681591b0 2023-10-06T01:41:56.753 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.747+0000 7f256d885700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25680a72e0 con 0x7f256815d930 2023-10-06T01:41:56.753 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.747+0000 7f25667fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f256815d560 0x7f25680a9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:56.753 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.747+0000 7f25667fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f256815d560 0x7f25680a9af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53824/0 (socket says 172.21.15.27:53824) 2023-10-06T01:41:56.753 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.747+0000 7f25667fc700 1 -- 172.21.15.27:0/2264384785 learned_addr learned my addr 172.21.15.27:0/2264384785 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:56.753 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.748+0000 7f2566ffd700 1 --2- 172.21.15.27:0/2264384785 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f25681591b0 0x7f2568149160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:56.753 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.748+0000 7f25677fe700 1 --2- 172.21.15.27:0/2264384785 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f256815d930 0x7f25680aa030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:56.753 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.748+0000 7f25667fc700 1 -- 172.21.15.27:0/2264384785 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f256815d930 msgr2=0x7f25680aa030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:56.753 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.748+0000 7f25667fc700 1 --2- 172.21.15.27:0/2264384785 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f256815d930 0x7f25680aa030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:56.753 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.748+0000 7f25667fc700 1 -- 172.21.15.27:0/2264384785 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f25681591b0 msgr2=0x7f2568149160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:56.753 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.748+0000 7f25667fc700 1 --2- 172.21.15.27:0/2264384785 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f25681591b0 0x7f2568149160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:56.754 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.748+0000 7f25667fc700 1 -- 172.21.15.27:0/2264384785 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2560009a20 con 0x7f256815d560 2023-10-06T01:41:56.754 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.748+0000 7f25667fc700 1 --2- 172.21.15.27:0/2264384785 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f256815d560 0x7f25680a9af0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f255000c5b0 tx=0x7f2550001070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:56.754 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.748+0000 7f255effd700 1 -- 172.21.15.27:0/2264384785 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2550005b30 con 0x7f256815d560 2023-10-06T01:41:56.754 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.748+0000 7f255effd700 1 -- 172.21.15.27:0/2264384785 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2550005c90 con 0x7f256815d560 2023-10-06T01:41:56.754 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.748+0000 7f256d885700 1 -- 172.21.15.27:0/2264384785 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2558009420 con 0x7f256815d560 2023-10-06T01:41:56.754 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.748+0000 7f255effd700 1 -- 172.21.15.27:0/2264384785 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2550007300 con 0x7f256815d560 2023-10-06T01:41:56.754 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.748+0000 7f256d885700 1 -- 172.21.15.27:0/2264384785 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25680a78a0 con 0x7f256815d560 2023-10-06T01:41:56.754 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.749+0000 7f256d885700 1 -- 172.21.15.27:0/2264384785 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2568064050 con 0x7f256815d560 2023-10-06T01:41:56.754 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.750+0000 7f255effd700 1 -- 172.21.15.27:0/2264384785 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f255001b3b0 con 0x7f256815d560 2023-10-06T01:41:56.754 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.750+0000 7f255effd700 1 --2- 172.21.15.27:0/2264384785 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2548066f30 0x7f25480693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:56.754 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.750+0000 7f255effd700 1 -- 172.21.15.27:0/2264384785 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f2550083770 con 0x7f256815d560 2023-10-06T01:41:56.755 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.750+0000 7f2566ffd700 1 --2- 172.21.15.27:0/2264384785 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2548066f30 0x7f25480693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:56.755 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.751+0000 7f2566ffd700 1 --2- 172.21.15.27:0/2264384785 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2548066f30 0x7f25480693e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f25680b19f0 tx=0x7f256000b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:56.755 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:56.754+0000 7f255effd700 1 -- 172.21.15.27:0/2264384785 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2550050bd0 con 0x7f256815d560 2023-10-06T01:41:57.029 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.029+0000 7f256d885700 1 -- 172.21.15.27:0/2264384785 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f25680a7d50 con 0x7f256815d560 2023-10-06T01:41:57.030 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.030+0000 7f255effd700 1 -- 172.21.15.27:0/2264384785 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v6) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f25500541f0 con 0x7f256815d560 2023-10-06T01:41:57.030 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:57.030 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2023-10-06T01:41:57.030 INFO:teuthology.orchestra.run.smithi027.stdout: "auth_dump": [ 2023-10-06T01:41:57.030 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:41:57.031 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.a", 2023-10-06T01:41:57.031 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCEZR9lILtQOxAAQI7GMolNknD5BJTrG23HmA==", 2023-10-06T01:41:57.031 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2023-10-06T01:41:57.031 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2023-10-06T01:41:57.031 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2023-10-06T01:41:57.031 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2023-10-06T01:41:57.031 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2023-10-06T01:41:57.031 INFO:teuthology.orchestra.run.smithi027.stdout: } 2023-10-06T01:41:57.031 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2023-10-06T01:41:57.031 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:41:57.031 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.b", 2023-10-06T01:41:57.031 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCFZR9liOgYCxAAZ1OTCiARNIWws/bd7jyOUg==", 2023-10-06T01:41:57.032 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2023-10-06T01:41:57.032 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2023-10-06T01:41:57.032 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2023-10-06T01:41:57.032 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2023-10-06T01:41:57.032 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2023-10-06T01:41:57.032 INFO:teuthology.orchestra.run.smithi027.stdout: } 2023-10-06T01:41:57.032 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2023-10-06T01:41:57.032 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:41:57.032 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.c", 2023-10-06T01:41:57.032 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCFZR9lLf+qBhAAufgkGJq1ycJiPAd2W6w/Jg==", 2023-10-06T01:41:57.032 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2023-10-06T01:41:57.032 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2023-10-06T01:41:57.032 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2023-10-06T01:41:57.033 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2023-10-06T01:41:57.033 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2023-10-06T01:41:57.033 INFO:teuthology.orchestra.run.smithi027.stdout: } 2023-10-06T01:41:57.033 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2023-10-06T01:41:57.033 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:41:57.033 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mds.d", 2023-10-06T01:41:57.033 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCFZR9lYybhERAAMmpeZMo/+odBszUSL8MTRQ==", 2023-10-06T01:41:57.033 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2023-10-06T01:41:57.033 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2023-10-06T01:41:57.033 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2023-10-06T01:41:57.033 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2023-10-06T01:41:57.033 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2023-10-06T01:41:57.034 INFO:teuthology.orchestra.run.smithi027.stdout: } 2023-10-06T01:41:57.034 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2023-10-06T01:41:57.034 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:41:57.034 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.0", 2023-10-06T01:41:57.034 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCIZR9lgBDDNxAArnRDoWNp3qAf8sQGLP62+g==", 2023-10-06T01:41:57.034 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2023-10-06T01:41:57.034 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2023-10-06T01:41:57.034 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2023-10-06T01:41:57.034 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2023-10-06T01:41:57.034 INFO:teuthology.orchestra.run.smithi027.stdout: } 2023-10-06T01:41:57.034 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2023-10-06T01:41:57.034 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:41:57.035 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.1", 2023-10-06T01:41:57.035 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCLZR9l3rBTNRAArTUFBhj89Mt4WXoLD6Zu4w==", 2023-10-06T01:41:57.035 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2023-10-06T01:41:57.035 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2023-10-06T01:41:57.035 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2023-10-06T01:41:57.035 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2023-10-06T01:41:57.035 INFO:teuthology.orchestra.run.smithi027.stdout: } 2023-10-06T01:41:57.035 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2023-10-06T01:41:57.035 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:41:57.035 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.2", 2023-10-06T01:41:57.035 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCOZR9lzz1iNxAAS9/IvCyrXwtoTEQAXbKIzw==", 2023-10-06T01:41:57.035 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2023-10-06T01:41:57.036 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2023-10-06T01:41:57.036 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2023-10-06T01:41:57.036 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2023-10-06T01:41:57.036 INFO:teuthology.orchestra.run.smithi027.stdout: } 2023-10-06T01:41:57.036 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2023-10-06T01:41:57.036 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:41:57.036 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.3", 2023-10-06T01:41:57.036 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCRZR9lv2HsNRAA1jUJVb5QhD3r0i0r5NZFSA==", 2023-10-06T01:41:57.036 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2023-10-06T01:41:57.036 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2023-10-06T01:41:57.036 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2023-10-06T01:41:57.036 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2023-10-06T01:41:57.037 INFO:teuthology.orchestra.run.smithi027.stdout: } 2023-10-06T01:41:57.037 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2023-10-06T01:41:57.037 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:41:57.037 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.4", 2023-10-06T01:41:57.037 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCXZR9ljvX0MRAAFdyhJLaMeFcjE4s3nZu2dg==", 2023-10-06T01:41:57.037 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2023-10-06T01:41:57.037 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2023-10-06T01:41:57.037 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2023-10-06T01:41:57.037 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2023-10-06T01:41:57.037 INFO:teuthology.orchestra.run.smithi027.stdout: } 2023-10-06T01:41:57.037 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2023-10-06T01:41:57.037 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:41:57.038 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.5", 2023-10-06T01:41:57.038 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCaZR9l9ErgLhAAF9++aR832vr6r/8K+MWHIA==", 2023-10-06T01:41:57.038 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2023-10-06T01:41:57.038 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2023-10-06T01:41:57.038 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2023-10-06T01:41:57.038 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2023-10-06T01:41:57.038 INFO:teuthology.orchestra.run.smithi027.stdout: } 2023-10-06T01:41:57.038 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2023-10-06T01:41:57.038 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:41:57.038 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.6", 2023-10-06T01:41:57.038 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCdZR9l8EpkLhAATNtUt3vTRMVpi7kHG2gACA==", 2023-10-06T01:41:57.038 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2023-10-06T01:41:57.038 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2023-10-06T01:41:57.039 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2023-10-06T01:41:57.039 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2023-10-06T01:41:57.039 INFO:teuthology.orchestra.run.smithi027.stdout: } 2023-10-06T01:41:57.039 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2023-10-06T01:41:57.039 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:41:57.039 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "osd.7", 2023-10-06T01:41:57.039 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCgZR9l0VspKxAAdD8MRmsSxkNE50yx+9RjZA==", 2023-10-06T01:41:57.039 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2023-10-06T01:41:57.039 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow profile osd", 2023-10-06T01:41:57.039 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile osd", 2023-10-06T01:41:57.039 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2023-10-06T01:41:57.039 INFO:teuthology.orchestra.run.smithi027.stdout: } 2023-10-06T01:41:57.040 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2023-10-06T01:41:57.040 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:41:57.040 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.0", 2023-10-06T01:41:57.040 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCFZR9lizfBFRAAp7lQ8h3aUs5zyzqT0vz2Kw==", 2023-10-06T01:41:57.040 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2023-10-06T01:41:57.040 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2023-10-06T01:41:57.040 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2023-10-06T01:41:57.040 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2023-10-06T01:41:57.040 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2023-10-06T01:41:57.040 INFO:teuthology.orchestra.run.smithi027.stdout: } 2023-10-06T01:41:57.040 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2023-10-06T01:41:57.040 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:41:57.040 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.1", 2023-10-06T01:41:57.041 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCFZR9lQAczHhAAxNf/t2BZLRUA1lbJqMLWDw==", 2023-10-06T01:41:57.041 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2023-10-06T01:41:57.041 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2023-10-06T01:41:57.041 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2023-10-06T01:41:57.041 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2023-10-06T01:41:57.041 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2023-10-06T01:41:57.041 INFO:teuthology.orchestra.run.smithi027.stdout: } 2023-10-06T01:41:57.041 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2023-10-06T01:41:57.041 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:41:57.041 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.2", 2023-10-06T01:41:57.041 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCFZR9lMwY1GxAAmB/1Xc/kVQZeDRJTW6Ca+w==", 2023-10-06T01:41:57.041 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2023-10-06T01:41:57.042 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2023-10-06T01:41:57.042 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2023-10-06T01:41:57.042 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2023-10-06T01:41:57.042 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2023-10-06T01:41:57.042 INFO:teuthology.orchestra.run.smithi027.stdout: } 2023-10-06T01:41:57.042 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2023-10-06T01:41:57.042 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:41:57.042 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.3", 2023-10-06T01:41:57.042 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCFZR9l7tOSIxAAxVuthAwmYrsU1HHI3ywf2g==", 2023-10-06T01:41:57.042 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2023-10-06T01:41:57.042 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow", 2023-10-06T01:41:57.042 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow r", 2023-10-06T01:41:57.043 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow rw", 2023-10-06T01:41:57.043 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow rwx" 2023-10-06T01:41:57.043 INFO:teuthology.orchestra.run.smithi027.stdout: } 2023-10-06T01:41:57.043 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2023-10-06T01:41:57.043 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:41:57.043 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.admin", 2023-10-06T01:41:57.043 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCEZR9lTXsZHxAArhpOotOKngHzPsog8q5drw==", 2023-10-06T01:41:57.043 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2023-10-06T01:41:57.043 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2023-10-06T01:41:57.043 INFO:teuthology.orchestra.run.smithi027.stdout: "mgr": "allow *", 2023-10-06T01:41:57.043 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow *", 2023-10-06T01:41:57.043 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2023-10-06T01:41:57.043 INFO:teuthology.orchestra.run.smithi027.stdout: } 2023-10-06T01:41:57.044 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2023-10-06T01:41:57.044 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:41:57.044 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.bootstrap-mds", 2023-10-06T01:41:57.044 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCmZR9llKlKHBAAQ6YyJoIuAGKAZK0FHpWFTw==", 2023-10-06T01:41:57.044 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2023-10-06T01:41:57.044 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile bootstrap-mds" 2023-10-06T01:41:57.044 INFO:teuthology.orchestra.run.smithi027.stdout: } 2023-10-06T01:41:57.044 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2023-10-06T01:41:57.044 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:41:57.044 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.bootstrap-mgr", 2023-10-06T01:41:57.044 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCmZR9l3bVKHBAAc77yckzWkdm21Ju8T0xegw==", 2023-10-06T01:41:57.044 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2023-10-06T01:41:57.045 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile bootstrap-mgr" 2023-10-06T01:41:57.045 INFO:teuthology.orchestra.run.smithi027.stdout: } 2023-10-06T01:41:57.045 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2023-10-06T01:41:57.045 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:41:57.045 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.bootstrap-osd", 2023-10-06T01:41:57.045 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCmZR9lvL5KHBAAXJep8qSPFWfQpGi8z2h4xA==", 2023-10-06T01:41:57.045 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2023-10-06T01:41:57.045 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile bootstrap-osd" 2023-10-06T01:41:57.045 INFO:teuthology.orchestra.run.smithi027.stdout: } 2023-10-06T01:41:57.045 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2023-10-06T01:41:57.045 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:41:57.045 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.bootstrap-rbd", 2023-10-06T01:41:57.045 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCmZR9lmsZKHBAAGw6v59aL8y5AFuPBCOtSeA==", 2023-10-06T01:41:57.046 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2023-10-06T01:41:57.046 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile bootstrap-rbd" 2023-10-06T01:41:57.046 INFO:teuthology.orchestra.run.smithi027.stdout: } 2023-10-06T01:41:57.046 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2023-10-06T01:41:57.046 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:41:57.046 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.bootstrap-rbd-mirror", 2023-10-06T01:41:57.046 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCmZR9lts5KHBAAmJJsj/pHeK7pOvM9KvW3ug==", 2023-10-06T01:41:57.046 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2023-10-06T01:41:57.046 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2023-10-06T01:41:57.046 INFO:teuthology.orchestra.run.smithi027.stdout: } 2023-10-06T01:41:57.046 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2023-10-06T01:41:57.046 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:41:57.047 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "client.bootstrap-rgw", 2023-10-06T01:41:57.047 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCmZR9l5tZKHBAAB6rREAfLECQGfrhvFTdcQg==", 2023-10-06T01:41:57.047 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2023-10-06T01:41:57.047 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile bootstrap-rgw" 2023-10-06T01:41:57.047 INFO:teuthology.orchestra.run.smithi027.stdout: } 2023-10-06T01:41:57.047 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2023-10-06T01:41:57.047 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:41:57.047 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.x", 2023-10-06T01:41:57.047 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCEZR9l5QQ1OBAAdjOKHB5rfu3W34oUlB8Wmw==", 2023-10-06T01:41:57.047 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2023-10-06T01:41:57.047 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2023-10-06T01:41:57.047 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2023-10-06T01:41:57.047 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2023-10-06T01:41:57.048 INFO:teuthology.orchestra.run.smithi027.stdout: } 2023-10-06T01:41:57.048 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2023-10-06T01:41:57.048 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:41:57.048 INFO:teuthology.orchestra.run.smithi027.stdout: "entity": "mgr.y", 2023-10-06T01:41:57.048 INFO:teuthology.orchestra.run.smithi027.stdout: "key": "AQCEZR9lV/5+MxAAaMkKkdnCzAPJAKk7KnEB0Q==", 2023-10-06T01:41:57.048 INFO:teuthology.orchestra.run.smithi027.stdout: "caps": { 2023-10-06T01:41:57.048 INFO:teuthology.orchestra.run.smithi027.stdout: "mds": "allow *", 2023-10-06T01:41:57.048 INFO:teuthology.orchestra.run.smithi027.stdout: "mon": "allow profile mgr", 2023-10-06T01:41:57.048 INFO:teuthology.orchestra.run.smithi027.stdout: "osd": "allow *" 2023-10-06T01:41:57.048 INFO:teuthology.orchestra.run.smithi027.stdout: } 2023-10-06T01:41:57.048 INFO:teuthology.orchestra.run.smithi027.stdout: } 2023-10-06T01:41:57.048 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2023-10-06T01:41:57.049 INFO:teuthology.orchestra.run.smithi027.stdout:} 2023-10-06T01:41:57.050 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.032+0000 7f256d885700 1 -- 172.21.15.27:0/2264384785 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2548066f30 msgr2=0x7f25480693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:57.050 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.032+0000 7f256d885700 1 --2- 172.21.15.27:0/2264384785 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2548066f30 0x7f25480693e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f25680b19f0 tx=0x7f256000b040 comp rx=0 tx=0).stop 2023-10-06T01:41:57.050 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.032+0000 7f256d885700 1 -- 172.21.15.27:0/2264384785 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f256815d560 msgr2=0x7f25680a9af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:57.050 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.032+0000 7f256d885700 1 --2- 172.21.15.27:0/2264384785 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f256815d560 0x7f25680a9af0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f255000c5b0 tx=0x7f2550001070 comp rx=0 tx=0).stop 2023-10-06T01:41:57.050 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.032+0000 7f256d885700 1 -- 172.21.15.27:0/2264384785 shutdown_connections 2023-10-06T01:41:57.050 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.032+0000 7f256d885700 1 --2- 172.21.15.27:0/2264384785 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2548066f30 0x7f25480693e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:57.050 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.032+0000 7f256d885700 1 --2- 172.21.15.27:0/2264384785 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f256815d930 0x7f25680aa030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:57.051 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.032+0000 7f256d885700 1 --2- 172.21.15.27:0/2264384785 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f25681591b0 0x7f2568149160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:57.051 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.032+0000 7f256d885700 1 --2- 172.21.15.27:0/2264384785 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f256815d560 0x7f25680a9af0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:57.051 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.032+0000 7f256d885700 1 -- 172.21.15.27:0/2264384785 >> 172.21.15.27:0/2264384785 conn(0x7f25680d1180 msgr2=0x7f25680b5480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:57.051 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.032+0000 7f256d885700 1 -- 172.21.15.27:0/2264384785 shutdown_connections 2023-10-06T01:41:57.051 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.032+0000 7f256d885700 1 -- 172.21.15.27:0/2264384785 wait complete. 2023-10-06T01:41:57.051 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2023-10-06T01:41:57.170 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.170+0000 7f4254847700 1 Processor -- start 2023-10-06T01:41:57.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.170+0000 7f4254847700 1 -- start start 2023-10-06T01:41:57.173 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.171+0000 7f4254847700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4250159500 0x7f42501598d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:57.173 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.171+0000 7f4254847700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f425015dab0 0x7f42500cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:57.173 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.171+0000 7f4254847700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42500b1110 con 0x7f42500b3fd0 2023-10-06T01:41:57.173 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.171+0000 7f4254847700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42500b13b0 con 0x7f425015dab0 2023-10-06T01:41:57.173 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.171+0000 7f4254847700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42500b1690 con 0x7f4250159500 2023-10-06T01:41:57.173 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.171+0000 7f424ed9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f425015dab0 0x7f42500cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:57.173 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.171+0000 7f424ed9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f425015dab0 0x7f42500cf6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53832/0 (socket says 172.21.15.27:53832) 2023-10-06T01:41:57.173 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.171+0000 7f424ed9d700 1 -- 172.21.15.27:0/3922446810 learned_addr learned my addr 172.21.15.27:0/3922446810 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:57.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.172+0000 7f424e59c700 1 --2- 172.21.15.27:0/3922446810 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4250159500 0x7f42501598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:57.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.172+0000 7f424ed9d700 1 -- 172.21.15.27:0/3922446810 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4250159500 msgr2=0x7f42501598d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:57.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.172+0000 7f424ed9d700 1 --2- 172.21.15.27:0/3922446810 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4250159500 0x7f42501598d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:57.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.172+0000 7f424ed9d700 1 -- 172.21.15.27:0/3922446810 >> v1:172.21.15.124:6789/0 conn(0x7f42500b3fd0 legacy=0x7f425015d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:57.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.172+0000 7f424ed9d700 1 -- 172.21.15.27:0/3922446810 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42500b1970 con 0x7f425015dab0 2023-10-06T01:41:57.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.172+0000 7f424ed9d700 1 --2- 172.21.15.27:0/3922446810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f425015dab0 0x7f42500cf6c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f4240009f20 tx=0x7f4240009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d069224f0af5be1a server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:57.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.172+0000 7f424d59a700 1 -- 172.21.15.27:0/3922446810 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f424000e070 con 0x7f425015dab0 2023-10-06T01:41:57.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.172+0000 7f424d59a700 1 -- 172.21.15.27:0/3922446810 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f424000ba80 con 0x7f425015dab0 2023-10-06T01:41:57.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.172+0000 7f424d59a700 1 -- 172.21.15.27:0/3922446810 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42400092e0 con 0x7f425015dab0 2023-10-06T01:41:57.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.173+0000 7f4254847700 1 -- 172.21.15.27:0/3922446810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f425015dab0 msgr2=0x7f42500cf6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:57.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.173+0000 7f4254847700 1 --2- 172.21.15.27:0/3922446810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f425015dab0 0x7f42500cf6c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f4240009f20 tx=0x7f4240009840 comp rx=0 tx=0).stop 2023-10-06T01:41:57.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.173+0000 7f4254847700 1 -- 172.21.15.27:0/3922446810 shutdown_connections 2023-10-06T01:41:57.175 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.173+0000 7f4254847700 1 --2- 172.21.15.27:0/3922446810 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f425015dab0 0x7f42500cf6c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:57.175 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.173+0000 7f4254847700 1 --2- 172.21.15.27:0/3922446810 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4250159500 0x7f42501598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:57.175 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.173+0000 7f4254847700 1 -- 172.21.15.27:0/3922446810 >> 172.21.15.27:0/3922446810 conn(0x7f4250067630 msgr2=0x7f4250069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:57.175 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.173+0000 7f4254847700 1 -- 172.21.15.27:0/3922446810 shutdown_connections 2023-10-06T01:41:57.175 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.173+0000 7f4254847700 1 -- 172.21.15.27:0/3922446810 wait complete. 2023-10-06T01:41:57.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.174+0000 7f4254847700 1 Processor -- start 2023-10-06T01:41:57.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.174+0000 7f4254847700 1 -- start start 2023-10-06T01:41:57.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.174+0000 7f4254847700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f42500b3fd0 0x7f425014db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:57.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.174+0000 7f4254847700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4250159500 0x7f425014e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:57.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.174+0000 7f4254847700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f425015dab0 0x7f425014ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:57.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.174+0000 7f4254847700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4250149050 con 0x7f425015dab0 2023-10-06T01:41:57.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.174+0000 7f4254847700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4250152670 con 0x7f4250159500 2023-10-06T01:41:57.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.174+0000 7f4254847700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f425014e040 con 0x7f42500b3fd0 2023-10-06T01:41:57.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.175+0000 7f424ed9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f425015dab0 0x7f425014ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:57.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.175+0000 7f424ed9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f425015dab0 0x7f425014ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53848/0 (socket says 172.21.15.27:53848) 2023-10-06T01:41:57.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.175+0000 7f424ed9d700 1 -- 172.21.15.27:0/754350309 learned_addr learned my addr 172.21.15.27:0/754350309 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:57.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.175+0000 7f424e59c700 1 --2- 172.21.15.27:0/754350309 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f42500b3fd0 0x7f425014db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:57.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.175+0000 7f424dd9b700 1 --2- 172.21.15.27:0/754350309 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4250159500 0x7f425014e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:57.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.175+0000 7f424ed9d700 1 -- 172.21.15.27:0/754350309 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f42500b3fd0 msgr2=0x7f425014db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:57.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.175+0000 7f424ed9d700 1 --2- 172.21.15.27:0/754350309 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f42500b3fd0 0x7f425014db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:57.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.175+0000 7f424ed9d700 1 -- 172.21.15.27:0/754350309 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4250159500 msgr2=0x7f425014e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:57.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.175+0000 7f424ed9d700 1 --2- 172.21.15.27:0/754350309 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4250159500 0x7f425014e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:57.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.175+0000 7f424ed9d700 1 -- 172.21.15.27:0/754350309 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4240009ae0 con 0x7f425015dab0 2023-10-06T01:41:57.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.175+0000 7f424ed9d700 1 --2- 172.21.15.27:0/754350309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f425015dab0 0x7f425014ef10 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4240012000 tx=0x7f4240018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:57.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.175+0000 7f42467fc700 1 -- 172.21.15.27:0/754350309 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42400032f0 con 0x7f425015dab0 2023-10-06T01:41:57.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.175+0000 7f42467fc700 1 -- 172.21.15.27:0/754350309 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4240008d20 con 0x7f425015dab0 2023-10-06T01:41:57.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.175+0000 7f42467fc700 1 -- 172.21.15.27:0/754350309 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f424001c640 con 0x7f425015dab0 2023-10-06T01:41:57.179 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.175+0000 7f4254847700 1 -- 172.21.15.27:0/754350309 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4250151a60 con 0x7f425015dab0 2023-10-06T01:41:57.179 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.175+0000 7f4254847700 1 -- 172.21.15.27:0/754350309 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4250152020 con 0x7f425015dab0 2023-10-06T01:41:57.179 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.176+0000 7f4254847700 1 -- 172.21.15.27:0/754350309 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4250152800 con 0x7f425015dab0 2023-10-06T01:41:57.179 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.177+0000 7f42467fc700 1 -- 172.21.15.27:0/754350309 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4240003040 con 0x7f425015dab0 2023-10-06T01:41:57.179 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.177+0000 7f42467fc700 1 --2- 172.21.15.27:0/754350309 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4230066f30 0x7f42300693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:57.180 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.177+0000 7f42467fc700 1 -- 172.21.15.27:0/754350309 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f4240085b70 con 0x7f425015dab0 2023-10-06T01:41:57.180 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.177+0000 7f424e59c700 1 --2- 172.21.15.27:0/754350309 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4230066f30 0x7f42300693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:57.180 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.178+0000 7f424e59c700 1 --2- 172.21.15.27:0/754350309 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4230066f30 0x7f42300693e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4248006fd0 tx=0x7f4248008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:57.181 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.181+0000 7f42467fc700 1 -- 172.21.15.27:0/754350309 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4240052fd0 con 0x7f425015dab0 2023-10-06T01:41:57.457 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.457+0000 7f4254847700 1 -- 172.21.15.27:0/754350309 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f4250064050 con 0x7f425015dab0 2023-10-06T01:41:57.461 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.461+0000 7f42467fc700 1 -- 172.21.15.27:0/754350309 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7f42400565f0 con 0x7f425015dab0 2023-10-06T01:41:57.463 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.463+0000 7f4254847700 1 -- 172.21.15.27:0/754350309 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4230066f30 msgr2=0x7f42300693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:57.463 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.463+0000 7f4254847700 1 --2- 172.21.15.27:0/754350309 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4230066f30 0x7f42300693e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4248006fd0 tx=0x7f4248008040 comp rx=0 tx=0).stop 2023-10-06T01:41:57.464 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.463+0000 7f4254847700 1 -- 172.21.15.27:0/754350309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f425015dab0 msgr2=0x7f425014ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:57.464 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.463+0000 7f4254847700 1 --2- 172.21.15.27:0/754350309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f425015dab0 0x7f425014ef10 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4240012000 tx=0x7f4240018000 comp rx=0 tx=0).stop 2023-10-06T01:41:57.464 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.463+0000 7f4254847700 1 -- 172.21.15.27:0/754350309 shutdown_connections 2023-10-06T01:41:57.464 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.463+0000 7f4254847700 1 --2- 172.21.15.27:0/754350309 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4230066f30 0x7f42300693e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:57.464 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.463+0000 7f4254847700 1 --2- 172.21.15.27:0/754350309 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f42500b3fd0 0x7f425014db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:57.464 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.463+0000 7f4254847700 1 --2- 172.21.15.27:0/754350309 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f425015dab0 0x7f425014ef10 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:57.464 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.463+0000 7f4254847700 1 --2- 172.21.15.27:0/754350309 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4250159500 0x7f425014e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:57.464 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.463+0000 7f4254847700 1 -- 172.21.15.27:0/754350309 >> 172.21.15.27:0/754350309 conn(0x7f4250067630 msgr2=0x7f42501492a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:57.464 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.463+0000 7f4254847700 1 -- 172.21.15.27:0/754350309 shutdown_connections 2023-10-06T01:41:57.464 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.463+0000 7f4254847700 1 -- 172.21.15.27:0/754350309 wait complete. 2023-10-06T01:41:57.475 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2023-10-06T01:41:57.595 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.594+0000 7f4b13a55700 1 Processor -- start 2023-10-06T01:41:57.595 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.594+0000 7f4b13a55700 1 -- start start 2023-10-06T01:41:57.597 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.595+0000 7f4b13a55700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4b0c157b50 0x7f4b0c0cd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:57.597 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.595+0000 7f4b13a55700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4b0c158630 0x7f4b0c161110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:57.597 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.595+0000 7f4b13a55700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b0c159a60 con 0x7f4b0c0cdb30 2023-10-06T01:41:57.597 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.595+0000 7f4b13a55700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b0c0cdfb0 con 0x7f4b0c157b50 2023-10-06T01:41:57.597 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.595+0000 7f4b13a55700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0c158e50 con 0x7f4b0c158630 2023-10-06T01:41:57.597 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.595+0000 7f4b117f1700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4b0c157b50 0x7f4b0c0cd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:57.597 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.595+0000 7f4b117f1700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4b0c157b50 0x7f4b0c0cd5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53854/0 (socket says 172.21.15.27:53854) 2023-10-06T01:41:57.598 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.595+0000 7f4b117f1700 1 -- 172.21.15.27:0/340028508 learned_addr learned my addr 172.21.15.27:0/340028508 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:57.598 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.595+0000 7f4b11ff2700 1 --2- 172.21.15.27:0/340028508 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4b0c158630 0x7f4b0c161110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:57.598 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.596+0000 7f4b117f1700 1 -- 172.21.15.27:0/340028508 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4b0c158630 msgr2=0x7f4b0c161110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:57.598 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.596+0000 7f4b117f1700 1 --2- 172.21.15.27:0/340028508 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4b0c158630 0x7f4b0c161110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:57.598 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.596+0000 7f4b117f1700 1 -- 172.21.15.27:0/340028508 >> v1:172.21.15.124:6789/0 conn(0x7f4b0c0cdb30 legacy=0x7f4b0c157f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:57.598 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.596+0000 7f4b117f1700 1 -- 172.21.15.27:0/340028508 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b0c159130 con 0x7f4b0c157b50 2023-10-06T01:41:57.598 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.596+0000 7f4b0bfff700 1 -- 172.21.15.27:0/340028508 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3478116042 0 0) 0x7f4b0c159a60 con 0x7f4b0c0cdb30 2023-10-06T01:41:57.598 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.596+0000 7f4b117f1700 1 --2- 172.21.15.27:0/340028508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b0c157b50 0x7f4b0c0cd5f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4b04009f20 tx=0x7f4b04009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f30b848c262bbfe8 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:57.598 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.596+0000 7f4b0bfff700 1 -- 172.21.15.27:0/340028508 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b0400e070 con 0x7f4b0c157b50 2023-10-06T01:41:57.598 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.596+0000 7f4b0bfff700 1 -- 172.21.15.27:0/340028508 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b0400b9d0 con 0x7f4b0c157b50 2023-10-06T01:41:57.598 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.596+0000 7f4b0bfff700 1 -- 172.21.15.27:0/340028508 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b040092e0 con 0x7f4b0c157b50 2023-10-06T01:41:57.599 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.597+0000 7f4b13a55700 1 -- 172.21.15.27:0/340028508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b0c157b50 msgr2=0x7f4b0c0cd5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:57.599 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.597+0000 7f4b13a55700 1 --2- 172.21.15.27:0/340028508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b0c157b50 0x7f4b0c0cd5f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4b04009f20 tx=0x7f4b04009840 comp rx=0 tx=0).stop 2023-10-06T01:41:57.599 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.597+0000 7f4b13a55700 1 -- 172.21.15.27:0/340028508 shutdown_connections 2023-10-06T01:41:57.599 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.597+0000 7f4b13a55700 1 --2- 172.21.15.27:0/340028508 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4b0c158630 0x7f4b0c161110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:57.599 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.597+0000 7f4b13a55700 1 --2- 172.21.15.27:0/340028508 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b0c157b50 0x7f4b0c0cd5f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:57.599 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.597+0000 7f4b13a55700 1 -- 172.21.15.27:0/340028508 >> 172.21.15.27:0/340028508 conn(0x7f4b0c06d890 msgr2=0x7f4b0c06dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:57.599 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.597+0000 7f4b13a55700 1 -- 172.21.15.27:0/340028508 shutdown_connections 2023-10-06T01:41:57.599 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.597+0000 7f4b13a55700 1 -- 172.21.15.27:0/340028508 wait complete. 2023-10-06T01:41:57.601 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.598+0000 7f4b13a55700 1 Processor -- start 2023-10-06T01:41:57.601 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.598+0000 7f4b13a55700 1 -- start start 2023-10-06T01:41:57.601 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.599+0000 7f4b13a55700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b0c0cdb30 0x7f4b0c140e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:57.601 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.599+0000 7f4b13a55700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4b0c157b50 0x7f4b0c06b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:57.602 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.599+0000 7f4b13a55700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4b0c158630 0x7f4b0c06b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:57.602 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.599+0000 7f4b13a55700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0c152c40 con 0x7f4b0c0cdb30 2023-10-06T01:41:57.602 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.599+0000 7f4b13a55700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0c152310 con 0x7f4b0c157b50 2023-10-06T01:41:57.602 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.599+0000 7f4b13a55700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0c1525f0 con 0x7f4b0c158630 2023-10-06T01:41:57.602 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.599+0000 7f4b117f1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b0c0cdb30 0x7f4b0c140e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:57.602 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.599+0000 7f4b117f1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b0c0cdb30 0x7f4b0c140e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53860/0 (socket says 172.21.15.27:53860) 2023-10-06T01:41:57.602 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.599+0000 7f4b117f1700 1 -- 172.21.15.27:0/3933386435 learned_addr learned my addr 172.21.15.27:0/3933386435 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:57.602 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.599+0000 7f4b11ff2700 1 --2- 172.21.15.27:0/3933386435 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4b0c158630 0x7f4b0c06b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:57.602 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.599+0000 7f4b10ff0700 1 --2- 172.21.15.27:0/3933386435 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4b0c157b50 0x7f4b0c06b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:57.602 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.599+0000 7f4b117f1700 1 -- 172.21.15.27:0/3933386435 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4b0c158630 msgr2=0x7f4b0c06b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:57.602 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.599+0000 7f4b117f1700 1 --2- 172.21.15.27:0/3933386435 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4b0c158630 0x7f4b0c06b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:57.602 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.599+0000 7f4b117f1700 1 -- 172.21.15.27:0/3933386435 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4b0c157b50 msgr2=0x7f4b0c06b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:57.603 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.599+0000 7f4b117f1700 1 --2- 172.21.15.27:0/3933386435 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4b0c157b50 0x7f4b0c06b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:57.603 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.599+0000 7f4b117f1700 1 -- 172.21.15.27:0/3933386435 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b04009ae0 con 0x7f4b0c0cdb30 2023-10-06T01:41:57.603 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.599+0000 7f4b10ff0700 1 --2- 172.21.15.27:0/3933386435 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4b0c157b50 0x7f4b0c06b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:41:57.603 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.599+0000 7f4b117f1700 1 --2- 172.21.15.27:0/3933386435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b0c0cdb30 0x7f4b0c140e50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c0afb20 tx=0x7f4b04018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:57.603 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.599+0000 7f4b097fa700 1 -- 172.21.15.27:0/3933386435 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b040089d0 con 0x7f4b0c0cdb30 2023-10-06T01:41:57.603 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.599+0000 7f4b097fa700 1 -- 172.21.15.27:0/3933386435 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b04008b30 con 0x7f4b0c0cdb30 2023-10-06T01:41:57.603 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.599+0000 7f4b13a55700 1 -- 172.21.15.27:0/3933386435 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b0c1528d0 con 0x7f4b0c0cdb30 2023-10-06T01:41:57.603 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.599+0000 7f4b097fa700 1 -- 172.21.15.27:0/3933386435 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b04026460 con 0x7f4b0c0cdb30 2023-10-06T01:41:57.603 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.600+0000 7f4b13a55700 1 -- 172.21.15.27:0/3933386435 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b0c154f80 con 0x7f4b0c0cdb30 2023-10-06T01:41:57.603 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.601+0000 7f4b13a55700 1 -- 172.21.15.27:0/3933386435 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b0c064050 con 0x7f4b0c0cdb30 2023-10-06T01:41:57.603 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.601+0000 7f4b097fa700 1 -- 172.21.15.27:0/3933386435 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4b04003040 con 0x7f4b0c0cdb30 2023-10-06T01:41:57.604 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.601+0000 7f4b097fa700 1 --2- 172.21.15.27:0/3933386435 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4af8066f30 0x7f4af80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:57.604 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.601+0000 7f4b097fa700 1 -- 172.21.15.27:0/3933386435 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f4b04086100 con 0x7f4b0c0cdb30 2023-10-06T01:41:57.604 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.602+0000 7f4b10ff0700 1 --2- 172.21.15.27:0/3933386435 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4af8066f30 0x7f4af80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:57.604 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.602+0000 7f4b10ff0700 1 --2- 172.21.15.27:0/3933386435 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4af8066f30 0x7f4af80693e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4b000011e0 tx=0x7f4b00009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:57.606 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.606+0000 7f4b097fa700 1 -- 172.21.15.27:0/3933386435 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4b04053610 con 0x7f4b0c0cdb30 2023-10-06T01:41:57.878 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.878+0000 7f4b13a55700 1 -- 172.21.15.27:0/3933386435 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f4b0c0cd4c0 con 0x7f4b0c0cdb30 2023-10-06T01:41:57.881 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.881+0000 7f4b097fa700 1 -- 172.21.15.27:0/3933386435 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f4b04056c30 con 0x7f4b0c0cdb30 2023-10-06T01:41:57.883 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.883+0000 7f4b13a55700 1 -- 172.21.15.27:0/3933386435 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4af8066f30 msgr2=0x7f4af80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:57.883 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.883+0000 7f4b13a55700 1 --2- 172.21.15.27:0/3933386435 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4af8066f30 0x7f4af80693e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4b000011e0 tx=0x7f4b00009040 comp rx=0 tx=0).stop 2023-10-06T01:41:57.884 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.883+0000 7f4b13a55700 1 -- 172.21.15.27:0/3933386435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b0c0cdb30 msgr2=0x7f4b0c140e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:57.884 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.883+0000 7f4b13a55700 1 --2- 172.21.15.27:0/3933386435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b0c0cdb30 0x7f4b0c140e50 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f4b0c0afb20 tx=0x7f4b04018000 comp rx=0 tx=0).stop 2023-10-06T01:41:57.884 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.883+0000 7f4b13a55700 1 -- 172.21.15.27:0/3933386435 shutdown_connections 2023-10-06T01:41:57.884 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.883+0000 7f4b13a55700 1 --2- 172.21.15.27:0/3933386435 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4af8066f30 0x7f4af80693e0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:57.884 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.883+0000 7f4b13a55700 1 --2- 172.21.15.27:0/3933386435 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4b0c158630 0x7f4b0c06b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:57.884 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.883+0000 7f4b13a55700 1 --2- 172.21.15.27:0/3933386435 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4b0c0cdb30 0x7f4b0c140e50 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:57.884 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.883+0000 7f4b13a55700 1 --2- 172.21.15.27:0/3933386435 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4b0c157b50 0x7f4b0c06b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:57.884 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.883+0000 7f4b13a55700 1 -- 172.21.15.27:0/3933386435 >> 172.21.15.27:0/3933386435 conn(0x7f4b0c06d890 msgr2=0x7f4b0c06ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:57.884 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.883+0000 7f4b13a55700 1 -- 172.21.15.27:0/3933386435 shutdown_connections 2023-10-06T01:41:57.884 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:57.883+0000 7f4b13a55700 1 -- 172.21.15.27:0/3933386435 wait complete. 2023-10-06T01:41:57.895 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2023-10-06T01:41:58.015 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.014+0000 7f091c852700 1 Processor -- start 2023-10-06T01:41:58.015 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.014+0000 7f091c852700 1 -- start start 2023-10-06T01:41:58.015 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.015+0000 7f091c852700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f091815dab0 0x7f09180cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:58.016 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.015+0000 7f091c852700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09180b1110 con 0x7f0918159500 2023-10-06T01:41:58.016 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.015+0000 7f091c852700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09180b13b0 con 0x7f09181598d0 2023-10-06T01:41:58.016 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.015+0000 7f091c852700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09180b1690 con 0x7f091815dab0 2023-10-06T01:41:58.016 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.015+0000 7f091659c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0918159500 0x7f09180b3fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54406/0 (socket says 172.21.15.27:54406) 2023-10-06T01:41:58.016 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.015+0000 7f091659c700 1 -- 172.21.15.27:0/3351224434 learned_addr learned my addr 172.21.15.27:0/3351224434 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:58.019 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.016+0000 7f0916d9d700 1 --2- 172.21.15.27:0/3351224434 >> v2:172.21.15.124:3300/0 conn(0x7f091815dab0 0x7f09180cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:58.019 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.016+0000 7f091559a700 1 -- 172.21.15.27:0/3351224434 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2508859490 0 0) 0x7f09180b1110 con 0x7f0918159500 2023-10-06T01:41:58.019 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.016+0000 7f091559a700 1 -- 172.21.15.27:0/3351224434 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0904003570 con 0x7f0918159500 2023-10-06T01:41:58.019 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.016+0000 7f091559a700 1 -- 172.21.15.27:0/3351224434 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1553465772 0 0) 0x7f0904003570 con 0x7f0918159500 2023-10-06T01:41:58.019 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.016+0000 7f091559a700 1 -- 172.21.15.27:0/3351224434 >> v2:172.21.15.124:3300/0 conn(0x7f091815dab0 msgr2=0x7f09180cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:58.019 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.016+0000 7f091559a700 1 --2- 172.21.15.27:0/3351224434 >> v2:172.21.15.124:3300/0 conn(0x7f091815dab0 0x7f09180cf6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:58.019 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.016+0000 7f091559a700 1 -- 172.21.15.27:0/3351224434 >> v1:172.21.15.124:6789/0 conn(0x7f09181598d0 legacy=0x7f091815d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:58.019 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.016+0000 7f091559a700 1 -- 172.21.15.27:0/3351224434 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09180b1970 con 0x7f0918159500 2023-10-06T01:41:58.019 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.016+0000 7f091559a700 1 -- 172.21.15.27:0/3351224434 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2619078944 0 0) 0x7f0910001470 con 0x7f0918159500 2023-10-06T01:41:58.020 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.016+0000 7f0916d9d700 1 --2- 172.21.15.27:0/3351224434 >> v2:172.21.15.124:3300/0 conn(0x7f091815dab0 0x7f09180cf6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:41:58.020 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.016+0000 7f091559a700 1 -- 172.21.15.27:0/3351224434 >> v1:172.21.15.27:6789/0 conn(0x7f0918159500 legacy=0x7f09180b3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:58.020 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.016+0000 7f091559a700 1 --2- 172.21.15.27:0/3351224434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0904003a60 0x7f0904003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:58.020 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.016+0000 7f091559a700 1 --2- 172.21.15.27:0/3351224434 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0904004810 0x7f0904004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:58.020 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.016+0000 7f091559a700 1 --2- 172.21.15.27:0/3351224434 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f09040051c0 0x7f090400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:58.020 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.016+0000 7f091559a700 1 -- 172.21.15.27:0/3351224434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09180b1690 con 0x7f0904003a60 2023-10-06T01:41:58.020 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.016+0000 7f091559a700 1 -- 172.21.15.27:0/3351224434 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f090400c0d0 con 0x7f0904004810 2023-10-06T01:41:58.020 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.016+0000 7f091559a700 1 -- 172.21.15.27:0/3351224434 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f090400c290 con 0x7f09040051c0 2023-10-06T01:41:58.020 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.016+0000 7f091659c700 1 --2- 172.21.15.27:0/3351224434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0904003a60 0x7f0904003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:58.020 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.016+0000 7f0915d9b700 1 --2- 172.21.15.27:0/3351224434 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0904004810 0x7f0904004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:58.020 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.016+0000 7f0916d9d700 1 --2- 172.21.15.27:0/3351224434 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f09040051c0 0x7f090400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:58.020 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.017+0000 7f091659c700 1 -- 172.21.15.27:0/3351224434 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f09040051c0 msgr2=0x7f090400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:58.021 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.017+0000 7f091659c700 1 --2- 172.21.15.27:0/3351224434 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f09040051c0 0x7f090400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:58.021 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.017+0000 7f091659c700 1 -- 172.21.15.27:0/3351224434 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0904004810 msgr2=0x7f0904004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:58.021 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.017+0000 7f091659c700 1 --2- 172.21.15.27:0/3351224434 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0904004810 0x7f0904004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:58.021 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.017+0000 7f091659c700 1 -- 172.21.15.27:0/3351224434 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0904003570 con 0x7f0904003a60 2023-10-06T01:41:58.021 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.017+0000 7f0916d9d700 1 --2- 172.21.15.27:0/3351224434 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f09040051c0 0x7f090400ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:58.021 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.017+0000 7f0916d9d700 1 -- 172.21.15.27:0/3351224434 reap_dead start 2023-10-06T01:41:58.021 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.017+0000 7f091659c700 1 --2- 172.21.15.27:0/3351224434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0904003a60 0x7f0904003e70 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f091000c3c0 tx=0x7f0910001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9295a44cf2574729 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:58.021 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.017+0000 7f091559a700 1 -- 172.21.15.27:0/3351224434 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f091000cc90 con 0x7f0904003a60 2023-10-06T01:41:58.021 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.017+0000 7f091559a700 1 -- 172.21.15.27:0/3351224434 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f091000cdf0 con 0x7f0904003a60 2023-10-06T01:41:58.021 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.017+0000 7f091559a700 1 -- 172.21.15.27:0/3351224434 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0910005b50 con 0x7f0904003a60 2023-10-06T01:41:58.021 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.017+0000 7f091c852700 1 -- 172.21.15.27:0/3351224434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0904003a60 msgr2=0x7f0904003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:58.021 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.017+0000 7f091c852700 1 --2- 172.21.15.27:0/3351224434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0904003a60 0x7f0904003e70 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f091000c3c0 tx=0x7f0910001e50 comp rx=0 tx=0).stop 2023-10-06T01:41:58.022 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.017+0000 7f091c852700 1 -- 172.21.15.27:0/3351224434 shutdown_connections 2023-10-06T01:41:58.022 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.017+0000 7f091c852700 1 --2- 172.21.15.27:0/3351224434 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0904003a60 0x7f0904003e70 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:58.022 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.017+0000 7f091c852700 1 -- 172.21.15.27:0/3351224434 >> 172.21.15.27:0/3351224434 conn(0x7f0918067630 msgr2=0x7f0918069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:58.022 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.017+0000 7f091c852700 1 -- 172.21.15.27:0/3351224434 shutdown_connections 2023-10-06T01:41:58.022 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.018+0000 7f091c852700 1 -- 172.21.15.27:0/3351224434 wait complete. 2023-10-06T01:41:58.022 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.019+0000 7f091c852700 1 Processor -- start 2023-10-06T01:41:58.022 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.019+0000 7f091c852700 1 -- start start 2023-10-06T01:41:58.024 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.019+0000 7f091c852700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0904003a60 0x7f091816d500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:58.024 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.019+0000 7f091c852700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f091813be50 0x7f091816da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:58.024 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.019+0000 7f091c852700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f091815a890 0x7f091815ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:58.025 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.019+0000 7f091c852700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091816ba00 con 0x7f091815a890 2023-10-06T01:41:58.025 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.019+0000 7f091c852700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f091815b5f0 con 0x7f091813be50 2023-10-06T01:41:58.025 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.019+0000 7f091c852700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f091816b0d0 con 0x7f0904003a60 2023-10-06T01:41:58.025 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.019+0000 7f0916d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f091815a890 0x7f091815ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:58.025 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.019+0000 7f0916d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f091815a890 0x7f091815ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53876/0 (socket says 172.21.15.27:53876) 2023-10-06T01:41:58.025 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.019+0000 7f0916d9d700 1 -- 172.21.15.27:0/334138349 learned_addr learned my addr 172.21.15.27:0/334138349 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:58.025 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.019+0000 7f091659c700 1 --2- 172.21.15.27:0/334138349 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0904003a60 0x7f091816d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:58.025 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.020+0000 7f0915d9b700 1 --2- 172.21.15.27:0/334138349 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f091813be50 0x7f091816da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:58.025 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.020+0000 7f0916d9d700 1 -- 172.21.15.27:0/334138349 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0904003a60 msgr2=0x7f091816d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:58.025 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.020+0000 7f0916d9d700 1 --2- 172.21.15.27:0/334138349 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0904003a60 0x7f091816d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:58.025 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.020+0000 7f0916d9d700 1 -- 172.21.15.27:0/334138349 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f091813be50 msgr2=0x7f091816da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:58.025 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.020+0000 7f0916d9d700 1 --2- 172.21.15.27:0/334138349 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f091813be50 0x7f091816da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:58.026 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.020+0000 7f0916d9d700 1 -- 172.21.15.27:0/334138349 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f091816b3b0 con 0x7f091815a890 2023-10-06T01:41:58.026 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.020+0000 7f0916d9d700 1 --2- 172.21.15.27:0/334138349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f091815a890 0x7f091815ad10 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f09180b2f00 tx=0x7f090800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:58.026 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.020+0000 7f090e7fc700 1 -- 172.21.15.27:0/334138349 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f090800e070 con 0x7f091815a890 2023-10-06T01:41:58.026 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.020+0000 7f090e7fc700 1 -- 172.21.15.27:0/334138349 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0908011c50 con 0x7f091815a890 2023-10-06T01:41:58.026 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.020+0000 7f090e7fc700 1 -- 172.21.15.27:0/334138349 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0908016470 con 0x7f091815a890 2023-10-06T01:41:58.026 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.020+0000 7f091c852700 1 -- 172.21.15.27:0/334138349 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f091816b690 con 0x7f091815a890 2023-10-06T01:41:58.026 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.020+0000 7f091c852700 1 -- 172.21.15.27:0/334138349 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09181522c0 con 0x7f091815a890 2023-10-06T01:41:58.026 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.021+0000 7f091c852700 1 -- 172.21.15.27:0/334138349 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f091816bbb0 con 0x7f091815a890 2023-10-06T01:41:58.026 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.022+0000 7f090e7fc700 1 -- 172.21.15.27:0/334138349 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0908016860 con 0x7f091815a890 2023-10-06T01:41:58.026 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.022+0000 7f090e7fc700 1 --2- 172.21.15.27:0/334138349 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f08f8066f30 0x7f08f80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:58.026 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.022+0000 7f090e7fc700 1 -- 172.21.15.27:0/334138349 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f0908084ee0 con 0x7f091815a890 2023-10-06T01:41:58.026 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.022+0000 7f091659c700 1 --2- 172.21.15.27:0/334138349 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f08f8066f30 0x7f08f80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:58.027 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.023+0000 7f091659c700 1 --2- 172.21.15.27:0/334138349 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f08f8066f30 0x7f08f80693e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f091815b9e0 tx=0x7f0910015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:58.027 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.026+0000 7f090e7fc700 1 -- 172.21.15.27:0/334138349 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f091816bbb0 con 0x7f091815a890 2023-10-06T01:41:58.296 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.296+0000 7f091c852700 1 -- 172.21.15.27:0/334138349 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f0918064050 con 0x7f091815a890 2023-10-06T01:41:58.301 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.301+0000 7f090e7fc700 1 -- 172.21.15.27:0/334138349 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v9) v1 ==== 90+0+0 (secure 0 0 0) 0x7f0908052340 con 0x7f091815a890 2023-10-06T01:41:58.303 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.303+0000 7f091c852700 1 -- 172.21.15.27:0/334138349 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f08f8066f30 msgr2=0x7f08f80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:58.303 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.303+0000 7f091c852700 1 --2- 172.21.15.27:0/334138349 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f08f8066f30 0x7f08f80693e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f091815b9e0 tx=0x7f0910015000 comp rx=0 tx=0).stop 2023-10-06T01:41:58.303 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.303+0000 7f091c852700 1 -- 172.21.15.27:0/334138349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f091815a890 msgr2=0x7f091815ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:58.303 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.303+0000 7f091c852700 1 --2- 172.21.15.27:0/334138349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f091815a890 0x7f091815ad10 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f09180b2f00 tx=0x7f090800b040 comp rx=0 tx=0).stop 2023-10-06T01:41:58.303 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.303+0000 7f091c852700 1 -- 172.21.15.27:0/334138349 shutdown_connections 2023-10-06T01:41:58.304 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.303+0000 7f091c852700 1 --2- 172.21.15.27:0/334138349 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f08f8066f30 0x7f08f80693e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:58.304 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.303+0000 7f091c852700 1 --2- 172.21.15.27:0/334138349 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0904003a60 0x7f091816d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:58.304 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.303+0000 7f091c852700 1 --2- 172.21.15.27:0/334138349 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f091815a890 0x7f091815ad10 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:58.304 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.303+0000 7f091c852700 1 --2- 172.21.15.27:0/334138349 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f091813be50 0x7f091816da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:58.304 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.303+0000 7f091c852700 1 -- 172.21.15.27:0/334138349 >> 172.21.15.27:0/334138349 conn(0x7f0918067630 msgr2=0x7f091815d2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:58.304 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.303+0000 7f091c852700 1 -- 172.21.15.27:0/334138349 shutdown_connections 2023-10-06T01:41:58.304 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.303+0000 7f091c852700 1 -- 172.21.15.27:0/334138349 wait complete. 2023-10-06T01:41:58.315 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2023-10-06T01:41:58.435 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.435+0000 7f8f5a610700 1 Processor -- start 2023-10-06T01:41:58.436 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.435+0000 7f8f5a610700 1 -- start start 2023-10-06T01:41:58.443 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.436+0000 7f8f5a610700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f8f54159500 0x7f8f541598d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:58.443 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.436+0000 7f8f5a610700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8f540b3fd0 0x7f8f5415d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:58.444 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.436+0000 7f8f5a610700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f5415dd20 con 0x7f8f5415d8e0 2023-10-06T01:41:58.444 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.436+0000 7f8f5a610700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f540b1110 con 0x7f8f540b3fd0 2023-10-06T01:41:58.444 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.436+0000 7f8f5a610700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f540b13b0 con 0x7f8f54159500 2023-10-06T01:41:58.444 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.436+0000 7f8f537fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8f540b3fd0 0x7f8f5415d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:58.444 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.436+0000 7f8f537fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8f540b3fd0 0x7f8f5415d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53886/0 (socket says 172.21.15.27:53886) 2023-10-06T01:41:58.445 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.436+0000 7f8f537fe700 1 -- 172.21.15.27:0/1539886145 learned_addr learned my addr 172.21.15.27:0/1539886145 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:58.445 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.436+0000 7f8f53fff700 1 --2- 172.21.15.27:0/1539886145 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f8f54159500 0x7f8f541598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:58.445 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.436+0000 7f8f537fe700 1 -- 172.21.15.27:0/1539886145 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f8f54159500 msgr2=0x7f8f541598d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:58.445 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.436+0000 7f8f537fe700 1 --2- 172.21.15.27:0/1539886145 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f8f54159500 0x7f8f541598d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:58.445 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.436+0000 7f8f537fe700 1 -- 172.21.15.27:0/1539886145 >> v1:172.21.15.124:6789/0 conn(0x7f8f5415d8e0 legacy=0x7f8f540cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:58.446 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.436+0000 7f8f537fe700 1 -- 172.21.15.27:0/1539886145 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f540b1690 con 0x7f8f540b3fd0 2023-10-06T01:41:58.446 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.436+0000 7f8f53fff700 1 --2- 172.21.15.27:0/1539886145 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f8f54159500 0x7f8f541598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:58.446 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.437+0000 7f8f537fe700 1 --2- 172.21.15.27:0/1539886145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f540b3fd0 0x7f8f5415d3a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f8f480099b0 tx=0x7f8f48009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2411e5119599e92 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:58.446 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.437+0000 7f8f52ffd700 1 -- 172.21.15.27:0/1539886145 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4800b860 con 0x7f8f540b3fd0 2023-10-06T01:41:58.446 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.437+0000 7f8f52ffd700 1 -- 172.21.15.27:0/1539886145 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f4800b9c0 con 0x7f8f540b3fd0 2023-10-06T01:41:58.446 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.437+0000 7f8f52ffd700 1 -- 172.21.15.27:0/1539886145 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4800bd60 con 0x7f8f540b3fd0 2023-10-06T01:41:58.447 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.437+0000 7f8f5a610700 1 -- 172.21.15.27:0/1539886145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f540b3fd0 msgr2=0x7f8f5415d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:58.447 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.437+0000 7f8f5a610700 1 --2- 172.21.15.27:0/1539886145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f540b3fd0 0x7f8f5415d3a0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f8f480099b0 tx=0x7f8f48009250 comp rx=0 tx=0).stop 2023-10-06T01:41:58.447 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.437+0000 7f8f5a610700 1 -- 172.21.15.27:0/1539886145 shutdown_connections 2023-10-06T01:41:58.447 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.437+0000 7f8f5a610700 1 --2- 172.21.15.27:0/1539886145 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f8f54159500 0x7f8f541598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:58.447 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.437+0000 7f8f5a610700 1 --2- 172.21.15.27:0/1539886145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f540b3fd0 0x7f8f5415d3a0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:58.447 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.437+0000 7f8f5a610700 1 -- 172.21.15.27:0/1539886145 >> 172.21.15.27:0/1539886145 conn(0x7f8f54067630 msgr2=0x7f8f54069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:58.447 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.437+0000 7f8f5a610700 1 -- 172.21.15.27:0/1539886145 shutdown_connections 2023-10-06T01:41:58.447 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.437+0000 7f8f5a610700 1 -- 172.21.15.27:0/1539886145 wait complete. 2023-10-06T01:41:58.448 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.438+0000 7f8f5a610700 1 Processor -- start 2023-10-06T01:41:58.448 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.439+0000 7f8f5a610700 1 -- start start 2023-10-06T01:41:58.448 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.439+0000 7f8f5a610700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f540b3fd0 0x7f8f5415bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:58.448 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.439+0000 7f8f5a610700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f8f54159500 0x7f8f541598d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:58.448 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.439+0000 7f8f5a610700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8f5415d8e0 0x7f8f5414a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:58.448 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.439+0000 7f8f5a610700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5415af10 con 0x7f8f540b3fd0 2023-10-06T01:41:58.448 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.439+0000 7f8f5a610700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5415a300 con 0x7f8f5415d8e0 2023-10-06T01:41:58.448 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.439+0000 7f8f5a610700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5415a5e0 con 0x7f8f54159500 2023-10-06T01:41:58.448 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.439+0000 7f8f53fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f540b3fd0 0x7f8f5415bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:58.449 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.439+0000 7f8f53fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f540b3fd0 0x7f8f5415bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53902/0 (socket says 172.21.15.27:53902) 2023-10-06T01:41:58.449 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.439+0000 7f8f53fff700 1 -- 172.21.15.27:0/1723620467 learned_addr learned my addr 172.21.15.27:0/1723620467 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:58.449 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.439+0000 7f8f58bad700 1 --2- 172.21.15.27:0/1723620467 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8f5415d8e0 0x7f8f5414a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:58.449 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.439+0000 7f8f537fe700 1 --2- 172.21.15.27:0/1723620467 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f8f54159500 0x7f8f541598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:58.449 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.440+0000 7f8f53fff700 1 -- 172.21.15.27:0/1723620467 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f8f54159500 msgr2=0x7f8f541598d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:58.449 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.440+0000 7f8f53fff700 1 --2- 172.21.15.27:0/1723620467 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f8f54159500 0x7f8f541598d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:58.449 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.440+0000 7f8f53fff700 1 -- 172.21.15.27:0/1723620467 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8f5415d8e0 msgr2=0x7f8f5414a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:58.449 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.440+0000 7f8f53fff700 1 --2- 172.21.15.27:0/1723620467 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8f5415d8e0 0x7f8f5414a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:58.449 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.440+0000 7f8f53fff700 1 -- 172.21.15.27:0/1723620467 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f480094f0 con 0x7f8f540b3fd0 2023-10-06T01:41:58.449 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.440+0000 7f8f53fff700 1 --2- 172.21.15.27:0/1723620467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f540b3fd0 0x7f8f5415bcb0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8f4400e8d0 tx=0x7f8f44008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:58.449 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.440+0000 7f8f3bfff700 1 -- 172.21.15.27:0/1723620467 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f44013410 con 0x7f8f540b3fd0 2023-10-06T01:41:58.449 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.440+0000 7f8f3bfff700 1 -- 172.21.15.27:0/1723620467 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f44013570 con 0x7f8f540b3fd0 2023-10-06T01:41:58.450 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.440+0000 7f8f5a610700 1 -- 172.21.15.27:0/1723620467 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f5415a8c0 con 0x7f8f540b3fd0 2023-10-06T01:41:58.450 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.440+0000 7f8f3bfff700 1 -- 172.21.15.27:0/1723620467 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f44013910 con 0x7f8f540b3fd0 2023-10-06T01:41:58.450 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.440+0000 7f8f5a610700 1 -- 172.21.15.27:0/1723620467 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f54069d80 con 0x7f8f540b3fd0 2023-10-06T01:41:58.450 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.441+0000 7f8f5a610700 1 -- 172.21.15.27:0/1723620467 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f5414dfe0 con 0x7f8f540b3fd0 2023-10-06T01:41:58.450 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.441+0000 7f8f3bfff700 1 -- 172.21.15.27:0/1723620467 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8f44013d00 con 0x7f8f540b3fd0 2023-10-06T01:41:58.450 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.442+0000 7f8f3bfff700 1 --2- 172.21.15.27:0/1723620467 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8f40066f30 0x7f8f400693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:58.450 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.442+0000 7f8f3bfff700 1 -- 172.21.15.27:0/1723620467 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f8f44084060 con 0x7f8f540b3fd0 2023-10-06T01:41:58.450 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.442+0000 7f8f537fe700 1 --2- 172.21.15.27:0/1723620467 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8f40066f30 0x7f8f400693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:58.450 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.442+0000 7f8f537fe700 1 --2- 172.21.15.27:0/1723620467 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8f40066f30 0x7f8f400693e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8f480099b0 tx=0x7f8f48017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:58.451 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.446+0000 7f8f3bfff700 1 -- 172.21.15.27:0/1723620467 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8f5414dfe0 con 0x7f8f540b3fd0 2023-10-06T01:41:58.718 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.718+0000 7f8f5a610700 1 -- 172.21.15.27:0/1723620467 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f8f5414dfe0 con 0x7f8f540b3fd0 2023-10-06T01:41:58.721 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.721+0000 7f8f3bfff700 1 -- 172.21.15.27:0/1723620467 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7f8f5414dfe0 con 0x7f8f540b3fd0 2023-10-06T01:41:58.724 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.723+0000 7f8f5a610700 1 -- 172.21.15.27:0/1723620467 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8f40066f30 msgr2=0x7f8f400693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:58.724 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.723+0000 7f8f5a610700 1 --2- 172.21.15.27:0/1723620467 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8f40066f30 0x7f8f400693e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8f480099b0 tx=0x7f8f48017000 comp rx=0 tx=0).stop 2023-10-06T01:41:58.724 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.723+0000 7f8f5a610700 1 -- 172.21.15.27:0/1723620467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f540b3fd0 msgr2=0x7f8f5415bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:58.724 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.723+0000 7f8f5a610700 1 --2- 172.21.15.27:0/1723620467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f540b3fd0 0x7f8f5415bcb0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8f4400e8d0 tx=0x7f8f44008040 comp rx=0 tx=0).stop 2023-10-06T01:41:58.724 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.723+0000 7f8f5a610700 1 -- 172.21.15.27:0/1723620467 shutdown_connections 2023-10-06T01:41:58.724 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.723+0000 7f8f5a610700 1 --2- 172.21.15.27:0/1723620467 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8f40066f30 0x7f8f400693e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:58.724 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.723+0000 7f8f5a610700 1 --2- 172.21.15.27:0/1723620467 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8f5415d8e0 0x7f8f5414a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:58.724 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.723+0000 7f8f5a610700 1 --2- 172.21.15.27:0/1723620467 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8f540b3fd0 0x7f8f5415bcb0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:58.724 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.723+0000 7f8f5a610700 1 --2- 172.21.15.27:0/1723620467 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f8f54159500 0x7f8f541598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:58.725 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.723+0000 7f8f5a610700 1 -- 172.21.15.27:0/1723620467 >> 172.21.15.27:0/1723620467 conn(0x7f8f54067630 msgr2=0x7f8f540b1d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:58.725 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.723+0000 7f8f5a610700 1 -- 172.21.15.27:0/1723620467 shutdown_connections 2023-10-06T01:41:58.725 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.723+0000 7f8f5a610700 1 -- 172.21.15.27:0/1723620467 wait complete. 2023-10-06T01:41:58.735 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2023-10-06T01:41:58.855 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.854+0000 7fb73e3bf700 1 Processor -- start 2023-10-06T01:41:58.855 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.854+0000 7fb73e3bf700 1 -- start start 2023-10-06T01:41:58.857 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.855+0000 7fb73e3bf700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb738159500 0x7fb7381598d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:58.857 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.855+0000 7fb73e3bf700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb7380b3fd0 0x7fb73815d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:58.857 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.855+0000 7fb73e3bf700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb73815dd20 con 0x7fb73815d8e0 2023-10-06T01:41:58.858 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.855+0000 7fb73e3bf700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7380b1110 con 0x7fb7380b3fd0 2023-10-06T01:41:58.858 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.855+0000 7fb73e3bf700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7380b13b0 con 0x7fb738159500 2023-10-06T01:41:58.858 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.855+0000 7fb7377fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb7380b3fd0 0x7fb73815d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:58.858 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.855+0000 7fb7377fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb7380b3fd0 0x7fb73815d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53912/0 (socket says 172.21.15.27:53912) 2023-10-06T01:41:58.858 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.855+0000 7fb7377fe700 1 -- 172.21.15.27:0/2374062165 learned_addr learned my addr 172.21.15.27:0/2374062165 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:58.858 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.855+0000 7fb737fff700 1 --2- 172.21.15.27:0/2374062165 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb738159500 0x7fb7381598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:58.858 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.856+0000 7fb7377fe700 1 -- 172.21.15.27:0/2374062165 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb738159500 msgr2=0x7fb7381598d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:58.859 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.856+0000 7fb7377fe700 1 --2- 172.21.15.27:0/2374062165 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb738159500 0x7fb7381598d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:58.859 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.856+0000 7fb7377fe700 1 -- 172.21.15.27:0/2374062165 >> v1:172.21.15.124:6789/0 conn(0x7fb73815d8e0 legacy=0x7fb7380cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:58.859 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.856+0000 7fb7377fe700 1 -- 172.21.15.27:0/2374062165 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7380b1690 con 0x7fb7380b3fd0 2023-10-06T01:41:58.859 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.856+0000 7fb7377fe700 1 --2- 172.21.15.27:0/2374062165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7380b3fd0 0x7fb73815d3a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb72c0099b0 tx=0x7fb72c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2472246df1a37a7b server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:58.859 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.856+0000 7fb736ffd700 1 -- 172.21.15.27:0/2374062165 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72c009ea0 con 0x7fb7380b3fd0 2023-10-06T01:41:58.859 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.856+0000 7fb736ffd700 1 -- 172.21.15.27:0/2374062165 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb72c00b860 con 0x7fb7380b3fd0 2023-10-06T01:41:58.860 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.856+0000 7fb736ffd700 1 -- 172.21.15.27:0/2374062165 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72c00bc00 con 0x7fb7380b3fd0 2023-10-06T01:41:58.860 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.857+0000 7fb73e3bf700 1 -- 172.21.15.27:0/2374062165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7380b3fd0 msgr2=0x7fb73815d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:58.860 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.857+0000 7fb73e3bf700 1 --2- 172.21.15.27:0/2374062165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7380b3fd0 0x7fb73815d3a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb72c0099b0 tx=0x7fb72c009250 comp rx=0 tx=0).stop 2023-10-06T01:41:58.860 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.857+0000 7fb73e3bf700 1 -- 172.21.15.27:0/2374062165 shutdown_connections 2023-10-06T01:41:58.860 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.857+0000 7fb73e3bf700 1 --2- 172.21.15.27:0/2374062165 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb738159500 0x7fb7381598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:58.860 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.857+0000 7fb73e3bf700 1 --2- 172.21.15.27:0/2374062165 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7380b3fd0 0x7fb73815d3a0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:58.860 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.857+0000 7fb73e3bf700 1 -- 172.21.15.27:0/2374062165 >> 172.21.15.27:0/2374062165 conn(0x7fb738067630 msgr2=0x7fb738069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:58.860 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.857+0000 7fb73e3bf700 1 -- 172.21.15.27:0/2374062165 shutdown_connections 2023-10-06T01:41:58.861 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.857+0000 7fb73e3bf700 1 -- 172.21.15.27:0/2374062165 wait complete. 2023-10-06T01:41:58.864 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.858+0000 7fb73e3bf700 1 Processor -- start 2023-10-06T01:41:58.864 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.858+0000 7fb73e3bf700 1 -- start start 2023-10-06T01:41:58.864 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.859+0000 7fb73e3bf700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7380b3fd0 0x7fb73814b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:58.864 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.859+0000 7fb73e3bf700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb738159500 0x7fb73816a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:58.864 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.859+0000 7fb73e3bf700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb73815d8e0 0x7fb73814d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:58.864 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.859+0000 7fb73e3bf700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73814ee70 con 0x7fb7380b3fd0 2023-10-06T01:41:58.864 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.859+0000 7fb73e3bf700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb73816b960 con 0x7fb73815d8e0 2023-10-06T01:41:58.864 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.859+0000 7fb73e3bf700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb73816ad40 con 0x7fb738159500 2023-10-06T01:41:58.864 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.859+0000 7fb737fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7380b3fd0 0x7fb73814b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:58.865 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.859+0000 7fb737fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7380b3fd0 0x7fb73814b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53920/0 (socket says 172.21.15.27:53920) 2023-10-06T01:41:58.865 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.859+0000 7fb737fff700 1 -- 172.21.15.27:0/1207382315 learned_addr learned my addr 172.21.15.27:0/1207382315 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:58.865 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.859+0000 7fb73c95c700 1 --2- 172.21.15.27:0/1207382315 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb73815d8e0 0x7fb73814d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:58.865 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.859+0000 7fb7377fe700 1 --2- 172.21.15.27:0/1207382315 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb738159500 0x7fb73816a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:58.865 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.859+0000 7fb737fff700 1 -- 172.21.15.27:0/1207382315 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb738159500 msgr2=0x7fb73816a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:58.865 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.859+0000 7fb737fff700 1 --2- 172.21.15.27:0/1207382315 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb738159500 0x7fb73816a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:58.865 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.859+0000 7fb737fff700 1 -- 172.21.15.27:0/1207382315 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb73815d8e0 msgr2=0x7fb73814d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:58.865 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.859+0000 7fb737fff700 1 --2- 172.21.15.27:0/1207382315 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb73815d8e0 0x7fb73814d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:58.865 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.859+0000 7fb737fff700 1 -- 172.21.15.27:0/1207382315 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72c0094f0 con 0x7fb7380b3fd0 2023-10-06T01:41:58.865 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.859+0000 7fb737fff700 1 --2- 172.21.15.27:0/1207382315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7380b3fd0 0x7fb73814b270 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fb72800ec80 tx=0x7fb7280091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:58.865 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.859+0000 7fb71ffff700 1 -- 172.21.15.27:0/1207382315 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72800c9d0 con 0x7fb7380b3fd0 2023-10-06T01:41:58.865 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.859+0000 7fb71ffff700 1 -- 172.21.15.27:0/1207382315 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb72800cb30 con 0x7fb7380b3fd0 2023-10-06T01:41:58.866 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.859+0000 7fb71ffff700 1 -- 172.21.15.27:0/1207382315 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb728013470 con 0x7fb7380b3fd0 2023-10-06T01:41:58.866 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.859+0000 7fb73e3bf700 1 -- 172.21.15.27:0/1207382315 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb73816b020 con 0x7fb7380b3fd0 2023-10-06T01:41:58.866 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.860+0000 7fb73e3bf700 1 -- 172.21.15.27:0/1207382315 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb738152fc0 con 0x7fb7380b3fd0 2023-10-06T01:41:58.866 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.860+0000 7fb73e3bf700 1 -- 172.21.15.27:0/1207382315 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb73816bc40 con 0x7fb7380b3fd0 2023-10-06T01:41:58.866 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.861+0000 7fb71ffff700 1 -- 172.21.15.27:0/1207382315 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb728013820 con 0x7fb7380b3fd0 2023-10-06T01:41:58.866 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.861+0000 7fb71ffff700 1 --2- 172.21.15.27:0/1207382315 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb724066f30 0x7fb7240693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:58.866 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.861+0000 7fb71ffff700 1 -- 172.21.15.27:0/1207382315 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fb7280846d0 con 0x7fb7380b3fd0 2023-10-06T01:41:58.866 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.861+0000 7fb7377fe700 1 --2- 172.21.15.27:0/1207382315 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb724066f30 0x7fb7240693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:58.866 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.862+0000 7fb7377fe700 1 --2- 172.21.15.27:0/1207382315 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb724066f30 0x7fb7240693e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb72c0099b0 tx=0x7fb72c017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:58.867 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:58.865+0000 7fb71ffff700 1 -- 172.21.15.27:0/1207382315 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb728051b30 con 0x7fb7380b3fd0 2023-10-06T01:41:59.140 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.140+0000 7fb73e3bf700 1 -- 172.21.15.27:0/1207382315 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7fb73816b450 con 0x7fb7380b3fd0 2023-10-06T01:41:59.144 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.144+0000 7fb71ffff700 1 -- 172.21.15.27:0/1207382315 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v11) v1 ==== 97+0+0 (secure 0 0 0) 0x7fb73816b450 con 0x7fb7380b3fd0 2023-10-06T01:41:59.146 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.146+0000 7fb73e3bf700 1 -- 172.21.15.27:0/1207382315 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb724066f30 msgr2=0x7fb7240693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:59.146 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.146+0000 7fb73e3bf700 1 --2- 172.21.15.27:0/1207382315 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb724066f30 0x7fb7240693e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb72c0099b0 tx=0x7fb72c017000 comp rx=0 tx=0).stop 2023-10-06T01:41:59.147 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.146+0000 7fb73e3bf700 1 -- 172.21.15.27:0/1207382315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7380b3fd0 msgr2=0x7fb73814b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:59.147 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.146+0000 7fb73e3bf700 1 --2- 172.21.15.27:0/1207382315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7380b3fd0 0x7fb73814b270 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fb72800ec80 tx=0x7fb7280091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:59.147 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.146+0000 7fb73e3bf700 1 -- 172.21.15.27:0/1207382315 shutdown_connections 2023-10-06T01:41:59.147 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.146+0000 7fb73e3bf700 1 --2- 172.21.15.27:0/1207382315 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb724066f30 0x7fb7240693e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:59.147 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.146+0000 7fb73e3bf700 1 --2- 172.21.15.27:0/1207382315 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb73815d8e0 0x7fb73814d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:59.147 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.146+0000 7fb73e3bf700 1 --2- 172.21.15.27:0/1207382315 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb7380b3fd0 0x7fb73814b270 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:59.147 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.146+0000 7fb73e3bf700 1 --2- 172.21.15.27:0/1207382315 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb738159500 0x7fb73816a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:59.147 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.146+0000 7fb73e3bf700 1 -- 172.21.15.27:0/1207382315 >> 172.21.15.27:0/1207382315 conn(0x7fb738067630 msgr2=0x7fb73815cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:59.147 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.146+0000 7fb73e3bf700 1 -- 172.21.15.27:0/1207382315 shutdown_connections 2023-10-06T01:41:59.148 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.146+0000 7fb73e3bf700 1 -- 172.21.15.27:0/1207382315 wait complete. 2023-10-06T01:41:59.158 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2023-10-06T01:41:59.278 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.277+0000 7f4fbd61b700 1 Processor -- start 2023-10-06T01:41:59.278 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.278+0000 7f4fbd61b700 1 -- start start 2023-10-06T01:41:59.283 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.278+0000 7f4fbd61b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4fb8159500 0x7f4fb81598d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:59.283 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.278+0000 7f4fbd61b700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4fb80b3fd0 0x7f4fb815d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:59.283 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.278+0000 7f4fbd61b700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fb815dd20 con 0x7f4fb815d8e0 2023-10-06T01:41:59.284 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.278+0000 7f4fbd61b700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fb80b1110 con 0x7f4fb8159500 2023-10-06T01:41:59.284 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.278+0000 7f4fbd61b700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb80b13b0 con 0x7f4fb80b3fd0 2023-10-06T01:41:59.284 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.278+0000 7f4fb6ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4fb8159500 0x7f4fb81598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:59.284 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.279+0000 7f4fb77fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4fb815d8e0 0x7f4fb80cf6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54414/0 (socket says 172.21.15.27:54414) 2023-10-06T01:41:59.284 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.279+0000 7f4fb6ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f4fb8159500 0x7f4fb81598d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53922/0 (socket says 172.21.15.27:53922) 2023-10-06T01:41:59.284 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.279+0000 7f4fb77fe700 1 -- 172.21.15.27:0/2311988570 learned_addr learned my addr 172.21.15.27:0/2311988570 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:41:59.284 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.279+0000 7f4fb67fc700 1 --2- 172.21.15.27:0/2311988570 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4fb80b3fd0 0x7f4fb815d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:59.284 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.279+0000 7f4fb6ffd700 1 -- 172.21.15.27:0/2311988570 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4fb80b3fd0 msgr2=0x7f4fb815d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:59.284 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.279+0000 7f4fb6ffd700 1 --2- 172.21.15.27:0/2311988570 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4fb80b3fd0 0x7f4fb815d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:59.284 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.279+0000 7f4fb6ffd700 1 -- 172.21.15.27:0/2311988570 >> v1:172.21.15.27:6789/0 conn(0x7f4fb815d8e0 legacy=0x7f4fb80cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:59.285 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.279+0000 7f4fb6ffd700 1 -- 172.21.15.27:0/2311988570 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fb80b1690 con 0x7f4fb8159500 2023-10-06T01:41:59.285 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.279+0000 7f4fb67fc700 1 --2- 172.21.15.27:0/2311988570 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4fb80b3fd0 0x7f4fb815d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:59.285 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.279+0000 7f4fb6ffd700 1 --2- 172.21.15.27:0/2311988570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fb8159500 0x7f4fb81598d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f4fb00098e0 tx=0x7f4fb00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83871512a4d6e14e server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:59.285 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.279+0000 7f4fb5ffb700 1 -- 172.21.15.27:0/2311988570 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb0009e80 con 0x7f4fb8159500 2023-10-06T01:41:59.285 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.279+0000 7f4fb5ffb700 1 -- 172.21.15.27:0/2311988570 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fb000b860 con 0x7f4fb8159500 2023-10-06T01:41:59.285 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.279+0000 7f4fb5ffb700 1 -- 172.21.15.27:0/2311988570 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb000bbe0 con 0x7f4fb8159500 2023-10-06T01:41:59.285 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.280+0000 7f4fbd61b700 1 -- 172.21.15.27:0/2311988570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fb8159500 msgr2=0x7f4fb81598d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:59.285 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.280+0000 7f4fbd61b700 1 --2- 172.21.15.27:0/2311988570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fb8159500 0x7f4fb81598d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f4fb00098e0 tx=0x7f4fb00091a0 comp rx=0 tx=0).stop 2023-10-06T01:41:59.285 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.280+0000 7f4fbd61b700 1 -- 172.21.15.27:0/2311988570 shutdown_connections 2023-10-06T01:41:59.285 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.280+0000 7f4fbd61b700 1 --2- 172.21.15.27:0/2311988570 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fb8159500 0x7f4fb81598d0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:59.285 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.280+0000 7f4fbd61b700 1 --2- 172.21.15.27:0/2311988570 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4fb80b3fd0 0x7f4fb815d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:59.286 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.280+0000 7f4fbd61b700 1 -- 172.21.15.27:0/2311988570 >> 172.21.15.27:0/2311988570 conn(0x7f4fb8067630 msgr2=0x7f4fb8069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:59.286 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.280+0000 7f4fbd61b700 1 -- 172.21.15.27:0/2311988570 shutdown_connections 2023-10-06T01:41:59.286 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.280+0000 7f4fbd61b700 1 -- 172.21.15.27:0/2311988570 wait complete. 2023-10-06T01:41:59.286 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.281+0000 7f4fbd61b700 1 Processor -- start 2023-10-06T01:41:59.286 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.281+0000 7f4fbd61b700 1 -- start start 2023-10-06T01:41:59.286 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.282+0000 7f4fbd61b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fb80b3fd0 0x7f4fb8155600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:59.286 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.282+0000 7f4fbd61b700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4fb8159500 0x7f4fb81509a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:59.286 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.282+0000 7f4fbd61b700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4fb815d8e0 0x7f4fb8150ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:59.286 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.282+0000 7f4fbd61b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb81514d0 con 0x7f4fb80b3fd0 2023-10-06T01:41:59.286 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.282+0000 7f4fbd61b700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb815a830 con 0x7f4fb815d8e0 2023-10-06T01:41:59.286 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.282+0000 7f4fbd61b700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb8159a70 con 0x7f4fb8159500 2023-10-06T01:41:59.287 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.282+0000 7f4fb6ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fb80b3fd0 0x7f4fb8155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:59.287 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.282+0000 7f4fb6ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fb80b3fd0 0x7f4fb8155600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53934/0 (socket says 172.21.15.27:53934) 2023-10-06T01:41:59.287 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.282+0000 7f4fb6ffd700 1 -- 172.21.15.27:0/2514291449 learned_addr learned my addr 172.21.15.27:0/2514291449 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:59.287 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.282+0000 7f4fb6ffd700 1 -- 172.21.15.27:0/2514291449 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4fb8159500 msgr2=0x7f4fb81509a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:59.287 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.282+0000 7f4fb6ffd700 1 --2- 172.21.15.27:0/2514291449 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4fb8159500 0x7f4fb81509a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:59.287 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.282+0000 7f4fb6ffd700 1 -- 172.21.15.27:0/2514291449 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4fb815d8e0 msgr2=0x7f4fb8150ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:59.287 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.282+0000 7f4fb77fe700 1 --2- 172.21.15.27:0/2514291449 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4fb815d8e0 0x7f4fb8150ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:59.287 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.282+0000 7f4fb6ffd700 1 --2- 172.21.15.27:0/2514291449 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4fb815d8e0 0x7f4fb8150ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:59.287 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.282+0000 7f4fb6ffd700 1 -- 172.21.15.27:0/2514291449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fb0009420 con 0x7f4fb80b3fd0 2023-10-06T01:41:59.287 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.282+0000 7f4fb6ffd700 1 --2- 172.21.15.27:0/2514291449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fb80b3fd0 0x7f4fb8155600 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4fb815deb0 tx=0x7f4fb0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:59.287 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.282+0000 7f4faeffd700 1 -- 172.21.15.27:0/2514291449 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb000bbe0 con 0x7f4fb80b3fd0 2023-10-06T01:41:59.288 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.282+0000 7f4faeffd700 1 -- 172.21.15.27:0/2514291449 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fb0011420 con 0x7f4fb80b3fd0 2023-10-06T01:41:59.288 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.283+0000 7f4fbd61b700 1 -- 172.21.15.27:0/2514291449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fb8159d50 con 0x7f4fb80b3fd0 2023-10-06T01:41:59.288 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.283+0000 7f4faeffd700 1 -- 172.21.15.27:0/2514291449 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fb00117c0 con 0x7f4fb80b3fd0 2023-10-06T01:41:59.288 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.283+0000 7f4fbd61b700 1 -- 172.21.15.27:0/2514291449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fb815a310 con 0x7f4fb80b3fd0 2023-10-06T01:41:59.289 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.284+0000 7f4fbd61b700 1 -- 172.21.15.27:0/2514291449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fb8159bb0 con 0x7f4fb80b3fd0 2023-10-06T01:41:59.289 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.284+0000 7f4faeffd700 1 -- 172.21.15.27:0/2514291449 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4fb0011bf0 con 0x7f4fb80b3fd0 2023-10-06T01:41:59.289 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.284+0000 7f4faeffd700 1 --2- 172.21.15.27:0/2514291449 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4f98066f30 0x7f4f980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:59.289 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.285+0000 7f4faeffd700 1 -- 172.21.15.27:0/2514291449 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f4fb0085d70 con 0x7f4fb80b3fd0 2023-10-06T01:41:59.289 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.285+0000 7f4fb67fc700 1 --2- 172.21.15.27:0/2514291449 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4f98066f30 0x7f4f980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:59.289 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.285+0000 7f4fb67fc700 1 --2- 172.21.15.27:0/2514291449 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4f98066f30 0x7f4f980693e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4fa0006fd0 tx=0x7f4fa0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:59.290 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.289+0000 7f4faeffd700 1 -- 172.21.15.27:0/2514291449 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4fb8159bb0 con 0x7f4fb80b3fd0 2023-10-06T01:41:59.561 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.561+0000 7f4fbd61b700 1 -- 172.21.15.27:0/2514291449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f4fb8159bb0 con 0x7f4fb80b3fd0 2023-10-06T01:41:59.564 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.564+0000 7f4faeffd700 1 -- 172.21.15.27:0/2514291449 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v12) v1 ==== 90+0+0 (secure 0 0 0) 0x7f4fb8159bb0 con 0x7f4fb80b3fd0 2023-10-06T01:41:59.567 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.566+0000 7f4fbd61b700 1 -- 172.21.15.27:0/2514291449 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4f98066f30 msgr2=0x7f4f980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:59.567 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.566+0000 7f4fbd61b700 1 --2- 172.21.15.27:0/2514291449 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4f98066f30 0x7f4f980693e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4fa0006fd0 tx=0x7f4fa0008040 comp rx=0 tx=0).stop 2023-10-06T01:41:59.567 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.566+0000 7f4fbd61b700 1 -- 172.21.15.27:0/2514291449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fb80b3fd0 msgr2=0x7f4fb8155600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:59.567 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.566+0000 7f4fbd61b700 1 --2- 172.21.15.27:0/2514291449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fb80b3fd0 0x7f4fb8155600 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4fb815deb0 tx=0x7f4fb0017000 comp rx=0 tx=0).stop 2023-10-06T01:41:59.567 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.566+0000 7f4fbd61b700 1 -- 172.21.15.27:0/2514291449 shutdown_connections 2023-10-06T01:41:59.567 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.566+0000 7f4fbd61b700 1 --2- 172.21.15.27:0/2514291449 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4f98066f30 0x7f4f980693e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:59.567 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.566+0000 7f4fbd61b700 1 --2- 172.21.15.27:0/2514291449 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4fb815d8e0 0x7f4fb8150ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:59.567 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.566+0000 7f4fbd61b700 1 --2- 172.21.15.27:0/2514291449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4fb80b3fd0 0x7f4fb8155600 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:59.567 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.566+0000 7f4fbd61b700 1 --2- 172.21.15.27:0/2514291449 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4fb8159500 0x7f4fb81509a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:59.568 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.566+0000 7f4fbd61b700 1 -- 172.21.15.27:0/2514291449 >> 172.21.15.27:0/2514291449 conn(0x7f4fb8067630 msgr2=0x7f4fb815d1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:59.568 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.566+0000 7f4fbd61b700 1 -- 172.21.15.27:0/2514291449 shutdown_connections 2023-10-06T01:41:59.568 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.566+0000 7f4fbd61b700 1 -- 172.21.15.27:0/2514291449 wait complete. 2023-10-06T01:41:59.579 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T01:41:59.698 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.698+0000 7f62c4c51700 1 Processor -- start 2023-10-06T01:41:59.698 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.698+0000 7f62c4c51700 1 -- start start 2023-10-06T01:41:59.701 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.699+0000 7f62c4c51700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f62c015d560 0x7f62c00d0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:59.701 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.699+0000 7f62c4c51700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f62c01591b0 0x7f62c0159630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:59.701 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.699+0000 7f62c4c51700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c00b2ad0 con 0x7f62c015d930 2023-10-06T01:41:59.701 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.699+0000 7f62c4c51700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62c00b4160 con 0x7f62c01591b0 2023-10-06T01:41:59.701 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.699+0000 7f62c4c51700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62c0159b70 con 0x7f62c015d560 2023-10-06T01:41:59.701 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.699+0000 7f62bed9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f62c01591b0 0x7f62c0159630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:59.701 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.699+0000 7f62bed9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f62c01591b0 0x7f62c0159630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53938/0 (socket says 172.21.15.27:53938) 2023-10-06T01:41:59.701 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.699+0000 7f62bdd9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f62c015d930 0x7f62c00b1a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54422/0 (socket says 172.21.15.27:54422) 2023-10-06T01:41:59.701 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.699+0000 7f62bed9d700 1 -- 172.21.15.27:0/3789134561 learned_addr learned my addr 172.21.15.27:0/3789134561 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:59.701 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.699+0000 7f62be59c700 1 --2- 172.21.15.27:0/3789134561 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f62c015d560 0x7f62c00d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:59.701 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.699+0000 7f62bed9d700 1 -- 172.21.15.27:0/3789134561 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f62c015d560 msgr2=0x7f62c00d0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:41:59.702 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.699+0000 7f62bed9d700 1 --2- 172.21.15.27:0/3789134561 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f62c015d560 0x7f62c00d0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:59.702 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.699+0000 7f62bed9d700 1 -- 172.21.15.27:0/3789134561 >> v1:172.21.15.27:6789/0 conn(0x7f62c015d930 legacy=0x7f62c00b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:59.702 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.699+0000 7f62bd59a700 1 -- 172.21.15.27:0/3789134561 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3553126466 0 0) 0x7f62c00b2ad0 con 0x7f62c015d930 2023-10-06T01:41:59.702 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.699+0000 7f62bed9d700 1 -- 172.21.15.27:0/3789134561 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c00b2230 con 0x7f62c01591b0 2023-10-06T01:41:59.702 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.699+0000 7f62be59c700 1 --2- 172.21.15.27:0/3789134561 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f62c015d560 0x7f62c00d0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:41:59.702 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.699+0000 7f62bed9d700 1 --2- 172.21.15.27:0/3789134561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62c01591b0 0x7f62c0159630 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f62b0009f20 tx=0x7f62b0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=683256a2e0e4273e server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:59.702 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.699+0000 7f62bd59a700 1 -- 172.21.15.27:0/3789134561 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b000e070 con 0x7f62c01591b0 2023-10-06T01:41:59.702 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.699+0000 7f62bd59a700 1 -- 172.21.15.27:0/3789134561 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b000ba80 con 0x7f62c01591b0 2023-10-06T01:41:59.702 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.699+0000 7f62bd59a700 1 -- 172.21.15.27:0/3789134561 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b00092e0 con 0x7f62c01591b0 2023-10-06T01:41:59.702 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.700+0000 7f62c4c51700 1 -- 172.21.15.27:0/3789134561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62c01591b0 msgr2=0x7f62c0159630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:59.702 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.700+0000 7f62c4c51700 1 --2- 172.21.15.27:0/3789134561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62c01591b0 0x7f62c0159630 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f62b0009f20 tx=0x7f62b0009840 comp rx=0 tx=0).stop 2023-10-06T01:41:59.702 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.700+0000 7f62c4c51700 1 -- 172.21.15.27:0/3789134561 shutdown_connections 2023-10-06T01:41:59.703 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.700+0000 7f62c4c51700 1 --2- 172.21.15.27:0/3789134561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62c01591b0 0x7f62c0159630 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:59.703 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.700+0000 7f62c4c51700 1 --2- 172.21.15.27:0/3789134561 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f62c015d560 0x7f62c00d0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:59.703 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.700+0000 7f62c4c51700 1 -- 172.21.15.27:0/3789134561 >> 172.21.15.27:0/3789134561 conn(0x7f62c00d1180 msgr2=0x7f62c0069640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:59.703 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.700+0000 7f62c4c51700 1 -- 172.21.15.27:0/3789134561 shutdown_connections 2023-10-06T01:41:59.703 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.700+0000 7f62c4c51700 1 -- 172.21.15.27:0/3789134561 wait complete. 2023-10-06T01:41:59.705 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.701+0000 7f62c4c51700 1 Processor -- start 2023-10-06T01:41:59.706 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.701+0000 7f62c4c51700 1 -- start start 2023-10-06T01:41:59.706 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.702+0000 7f62c4c51700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f62c01591b0 0x7f62c015c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:59.706 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.702+0000 7f62c4c51700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f62c015d560 0x7f62c014f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:59.706 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.702+0000 7f62c4c51700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62c015d930 0x7f62c014f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:59.706 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.702+0000 7f62c4c51700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c0159d40 con 0x7f62c015d930 2023-10-06T01:41:59.707 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.702+0000 7f62c4c51700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c016d2b0 con 0x7f62c01591b0 2023-10-06T01:41:59.707 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.702+0000 7f62c4c51700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62c016d590 con 0x7f62c015d560 2023-10-06T01:41:59.707 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.702+0000 7f62bed9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62c015d930 0x7f62c014f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:59.707 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.702+0000 7f62bed9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62c015d930 0x7f62c014f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53942/0 (socket says 172.21.15.27:53942) 2023-10-06T01:41:59.707 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.702+0000 7f62bed9d700 1 -- 172.21.15.27:0/1583719625 learned_addr learned my addr 172.21.15.27:0/1583719625 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:41:59.707 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.702+0000 7f62be59c700 1 --2- 172.21.15.27:0/1583719625 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f62c01591b0 0x7f62c015c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:59.707 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.702+0000 7f62bdd9b700 1 --2- 172.21.15.27:0/1583719625 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f62c015d560 0x7f62c014f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:59.707 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.702+0000 7f62bed9d700 1 -- 172.21.15.27:0/1583719625 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f62c015d560 msgr2=0x7f62c014f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:59.707 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.702+0000 7f62bed9d700 1 --2- 172.21.15.27:0/1583719625 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f62c015d560 0x7f62c014f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:59.707 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.702+0000 7f62bed9d700 1 -- 172.21.15.27:0/1583719625 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f62c01591b0 msgr2=0x7f62c015c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:59.707 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.702+0000 7f62bed9d700 1 --2- 172.21.15.27:0/1583719625 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f62c01591b0 0x7f62c015c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:59.707 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.702+0000 7f62bed9d700 1 -- 172.21.15.27:0/1583719625 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b0009ae0 con 0x7f62c015d930 2023-10-06T01:41:59.708 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.702+0000 7f62bdd9b700 1 --2- 172.21.15.27:0/1583719625 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f62c015d560 0x7f62c014f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:41:59.708 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.703+0000 7f62bed9d700 1 --2- 172.21.15.27:0/1583719625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62c015d930 0x7f62c014f770 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f62b0009880 tx=0x7f62b0018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:59.708 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.703+0000 7f62b67fc700 1 -- 172.21.15.27:0/1583719625 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b00032f0 con 0x7f62c015d930 2023-10-06T01:41:59.708 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.703+0000 7f62b67fc700 1 -- 172.21.15.27:0/1583719625 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b0008e10 con 0x7f62c015d930 2023-10-06T01:41:59.708 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.703+0000 7f62c4c51700 1 -- 172.21.15.27:0/1583719625 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62c016d870 con 0x7f62c015d930 2023-10-06T01:41:59.708 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.703+0000 7f62b67fc700 1 -- 172.21.15.27:0/1583719625 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b00266b0 con 0x7f62c015d930 2023-10-06T01:41:59.708 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.703+0000 7f62c4c51700 1 -- 172.21.15.27:0/1583719625 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62c01504d0 con 0x7f62c015d930 2023-10-06T01:41:59.708 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.704+0000 7f62c4c51700 1 -- 172.21.15.27:0/1583719625 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62c0064050 con 0x7f62c015d930 2023-10-06T01:41:59.708 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.704+0000 7f62b67fc700 1 -- 172.21.15.27:0/1583719625 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f62b0003040 con 0x7f62c015d930 2023-10-06T01:41:59.708 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.704+0000 7f62b67fc700 1 --2- 172.21.15.27:0/1583719625 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f62a0066f30 0x7f62a00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:41:59.708 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.705+0000 7f62b67fc700 1 -- 172.21.15.27:0/1583719625 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f62b0086400 con 0x7f62c015d930 2023-10-06T01:41:59.708 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.705+0000 7f62be59c700 1 --2- 172.21.15.27:0/1583719625 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f62a0066f30 0x7f62a00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:41:59.709 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.705+0000 7f62be59c700 1 --2- 172.21.15.27:0/1583719625 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f62a0066f30 0x7f62a00693e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f62b8006f00 tx=0x7f62b8008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:41:59.709 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.709+0000 7f62b67fc700 1 -- 172.21.15.27:0/1583719625 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f62b0053860 con 0x7f62c015d930 2023-10-06T01:41:59.904 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.903+0000 7f62c4c51700 1 -- 172.21.15.27:0/1583719625 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62c00b2510 con 0x7f62c015d930 2023-10-06T01:41:59.904 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.904+0000 7f62b67fc700 1 -- 172.21.15.27:0/1583719625 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+13650 (secure 0 0 0) 0x7f62b0056e80 con 0x7f62c015d930 2023-10-06T01:41:59.905 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:41:59.905 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":27,"fsid":"20a7bff0-74f8-4f2f-b6c6-899e804bc965","created":"2023-10-06T01:40:54.474215+0000","modified":"2023-10-06T01:41:56.619354+0000","last_up_change":"2023-10-06T01:41:05.242847+0000","last_in_change":"2023-10-06T01:40:56.890960+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-10-06T01:41:05.685386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-10-06T01:41:06.092995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b902c86d-800c-40c5-b152-b545e50d02c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":90644},{"type":"v1","addr":"172.21.15.27:6801","nonce":90644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":90644},{"type":"v1","addr":"172.21.15.27:6803","nonce":90644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":90644},{"type":"v1","addr":"172.21.15.27:6807","nonce":90644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":90644},{"type":"v1","addr":"172.21.15.27:6805","nonce":90644}]},"public_addr":"172.21.15.27:6801/90644","cluster_addr":"172.21.15.27:6803/90644","heartbeat_back_addr":"172.21.15.27:6807/90644","heartbeat_front_addr":"172.21.15.27:6805/90644","state":["exists","up"]},{"osd":1,"uuid":"5b4a70e5-b06e-48cd-bfc4-7962c5d05386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":90643},{"type":"v1","addr":"172.21.15.27:6809","nonce":90643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":90643},{"type":"v1","addr":"172.21.15.27:6811","nonce":90643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":90643},{"type":"v1","addr":"172.21.15.27:6815","nonce":90643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":90643},{"type":"v1","addr":"172.21.15.27:6813","nonce":90643}]},"public_addr":"172.21.15.27:6809/90643","cluster_addr":"172.21.15.27:6811/90643","heartbeat_back_addr":"172.21.15.27:6815/90643","heartbeat_front_addr":"172.21.15.27:6813/90643","state":["exists","up"]},{"osd":2,"uuid":"22d9de46-4073-4336-9122-5227e601eca0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":90645},{"type":"v1","addr":"172.21.15.27:6817","nonce":90645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":90645},{"type":"v1","addr":"172.21.15.27:6819","nonce":90645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":90645},{"type":"v1","addr":"172.21.15.27:6823","nonce":90645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":90645},{"type":"v1","addr":"172.21.15.27:6821","nonce":90645}]},"public_addr":"172.21.15.27:6817/90645","cluster_addr":"172.21.15.27:6819/90645","heartbeat_back_addr":"172.21.15.27:6823/90645","heartbeat_front_addr":"172.21.15.27:6821/90645","state":["exists","up"]},{"osd":3,"uuid":"438bb2d3-34e2-4b7c-9734-8360c1b67650","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":90668},{"type":"v1","addr":"172.21.15.27:6825","nonce":90668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":90668},{"type":"v1","addr":"172.21.15.27:6827","nonce":90668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":90668},{"type":"v1","addr":"172.21.15.27:6831","nonce":90668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":90668},{"type":"v1","addr":"172.21.15.27:6829","nonce":90668}]},"public_addr":"172.21.15.27:6825/90668","cluster_addr":"172.21.15.27:6827/90668","heartbeat_back_addr":"172.21.15.27:6831/90668","heartbeat_front_addr":"172.21.15.27:6829/90668","state":["exists","up"]},{"osd":4,"uuid":"e1956a8e-e0f8-4e7c-acbb-40037415a354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":90863},{"type":"v1","addr":"172.21.15.124:6801","nonce":90863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":90863},{"type":"v1","addr":"172.21.15.124:6803","nonce":90863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":90863},{"type":"v1","addr":"172.21.15.124:6807","nonce":90863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":90863},{"type":"v1","addr":"172.21.15.124:6805","nonce":90863}]},"public_addr":"172.21.15.124:6801/90863","cluster_addr":"172.21.15.124:6803/90863","heartbeat_back_addr":"172.21.15.124:6807/90863","heartbeat_front_addr":"172.21.15.124:6805/90863","state":["exists","up"]},{"osd":5,"uuid":"306c59d4-846d-4b73-b1c9-051bec9e9368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":90885},{"type":"v1","addr":"172.21.15.124:6817","nonce":90885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":90885},{"type":"v1","addr":"172.21.15.124:6819","nonce":90885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":90885},{"type":"v1","addr":"172.21.15.124:6823","nonce":90885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":90885},{"type":"v1","addr":"172.21.15.124:6821","nonce":90885}]},"public_addr":"172.21.15.124:6817/90885","cluster_addr":"172.21.15.124:6819/90885","heartbeat_back_addr":"172.21.15.124:6823/90885","heartbeat_front_addr":"172.21.15.124:6821/90885","state":["exists","up"]},{"osd":6,"uuid":"15463e53-ac8d-47d1-b4f6-c0d31f29e4a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":90864},{"type":"v1","addr":"172.21.15.124:6809","nonce":90864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":90864},{"type":"v1","addr":"172.21.15.124:6811","nonce":90864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":90864},{"type":"v1","addr":"172.21.15.124:6815","nonce":90864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":90864},{"type":"v1","addr":"172.21.15.124:6813","nonce":90864}]},"public_addr":"172.21.15.124:6809/90864","cluster_addr":"172.21.15.124:6811/90864","heartbeat_back_addr":"172.21.15.124:6815/90864","heartbeat_front_addr":"172.21.15.124:6813/90864","state":["exists","up"]},{"osd":7,"uuid":"ae463a90-a9d3-498f-86fa-47a35b3fe889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":90894},{"type":"v1","addr":"172.21.15.124:6825","nonce":90894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":90894},{"type":"v1","addr":"172.21.15.124:6827","nonce":90894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":90894},{"type":"v1","addr":"172.21.15.124:6831","nonce":90894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":90894},{"type":"v1","addr":"172.21.15.124:6829","nonce":90894}]},"public_addr":"172.21.15.124:6825/90894","cluster_addr":"172.21.15.124:6827/90894","heartbeat_back_addr":"172.21.15.124:6831/90894","heartbeat_front_addr":"172.21.15.124:6829/90894","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.827704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.839951+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.032230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.050605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.987908+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.031290+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.080707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.053692+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T01:41:59.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.906+0000 7f62c4c51700 1 -- 172.21.15.27:0/1583719625 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f62a0066f30 msgr2=0x7f62a00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:59.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.906+0000 7f62c4c51700 1 --2- 172.21.15.27:0/1583719625 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f62a0066f30 0x7f62a00693e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f62b8006f00 tx=0x7f62b8008040 comp rx=0 tx=0).stop 2023-10-06T01:41:59.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.906+0000 7f62c4c51700 1 -- 172.21.15.27:0/1583719625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62c015d930 msgr2=0x7f62c014f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:41:59.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.906+0000 7f62c4c51700 1 --2- 172.21.15.27:0/1583719625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62c015d930 0x7f62c014f770 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f62b0009880 tx=0x7f62b0018000 comp rx=0 tx=0).stop 2023-10-06T01:41:59.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.906+0000 7f62c4c51700 1 -- 172.21.15.27:0/1583719625 shutdown_connections 2023-10-06T01:41:59.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.906+0000 7f62c4c51700 1 --2- 172.21.15.27:0/1583719625 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f62a0066f30 0x7f62a00693e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:59.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.906+0000 7f62c4c51700 1 --2- 172.21.15.27:0/1583719625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62c015d930 0x7f62c014f770 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:59.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.906+0000 7f62c4c51700 1 --2- 172.21.15.27:0/1583719625 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f62c01591b0 0x7f62c015c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:59.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.906+0000 7f62c4c51700 1 --2- 172.21.15.27:0/1583719625 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f62c015d560 0x7f62c014f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:41:59.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.906+0000 7f62c4c51700 1 -- 172.21.15.27:0/1583719625 >> 172.21.15.27:0/1583719625 conn(0x7f62c00d1180 msgr2=0x7f62c0067680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:41:59.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.906+0000 7f62c4c51700 1 -- 172.21.15.27:0/1583719625 shutdown_connections 2023-10-06T01:41:59.908 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:41:59.906+0000 7f62c4c51700 1 -- 172.21.15.27:0/1583719625 wait complete. 2023-10-06T01:41:59.919 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-10-06T01:41:05.685386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-10-06T01:41:06.092995+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-10-06T01:41:59.919 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-06T01:42:00.040 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.040+0000 7f69a1977700 1 Processor -- start 2023-10-06T01:42:00.040 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.040+0000 7f69a1977700 1 -- start start 2023-10-06T01:42:00.042 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.041+0000 7f69a1977700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f699c1593f0 0x7f699c0b00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:00.042 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.041+0000 7f69a1977700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f699c1597c0 0x7f699c0cf680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:00.042 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.041+0000 7f69a1977700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f699c0cfbc0 0x7f699c068350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:00.042 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.041+0000 7f69a1977700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f699c0cab50 con 0x7f699c1597c0 2023-10-06T01:42:00.042 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.041+0000 7f69a1977700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f699c0cb3f0 con 0x7f699c0cfbc0 2023-10-06T01:42:00.042 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.041+0000 7f69a1977700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f699c0b05f0 con 0x7f699c1593f0 2023-10-06T01:42:00.042 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.041+0000 7f699bfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f699c1597c0 0x7f699c0cf680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:00.042 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.041+0000 7f699bfff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f699c1597c0 0x7f699c0cf680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53950/0 (socket says 172.21.15.27:53950) 2023-10-06T01:42:00.043 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.041+0000 7f69a0975700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f699c1593f0 0x7f699c0b00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:00.043 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.041+0000 7f699bfff700 1 -- 172.21.15.27:0/2909498579 learned_addr learned my addr 172.21.15.27:0/2909498579 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:00.043 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.041+0000 7f69a1176700 1 --2- 172.21.15.27:0/2909498579 >> v2:172.21.15.124:3300/0 conn(0x7f699c0cfbc0 0x7f699c068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:00.043 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.041+0000 7f699bfff700 1 -- 172.21.15.27:0/2909498579 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f699c1593f0 msgr2=0x7f699c0b00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:00.043 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.041+0000 7f699bfff700 1 --2- 172.21.15.27:0/2909498579 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f699c1593f0 0x7f699c0b00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:00.043 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.041+0000 7f699bfff700 1 -- 172.21.15.27:0/2909498579 >> v2:172.21.15.124:3300/0 conn(0x7f699c0cfbc0 msgr2=0x7f699c068350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:00.043 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.041+0000 7f699bfff700 1 --2- 172.21.15.27:0/2909498579 >> v2:172.21.15.124:3300/0 conn(0x7f699c0cfbc0 0x7f699c068350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:00.043 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.041+0000 7f699bfff700 1 -- 172.21.15.27:0/2909498579 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f699c0d0150 con 0x7f699c1597c0 2023-10-06T01:42:00.043 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.042+0000 7f699bfff700 1 --2- 172.21.15.27:0/2909498579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f699c1597c0 0x7f699c0cf680 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f699c06a080 tx=0x7f6988009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=677f919bb2645bdb server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:00.043 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.042+0000 7f699b7fe700 1 -- 172.21.15.27:0/2909498579 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698800b640 con 0x7f699c1597c0 2023-10-06T01:42:00.043 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.042+0000 7f699b7fe700 1 -- 172.21.15.27:0/2909498579 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f698800b7a0 con 0x7f699c1597c0 2023-10-06T01:42:00.043 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.042+0000 7f699b7fe700 1 -- 172.21.15.27:0/2909498579 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698800bb60 con 0x7f699c1597c0 2023-10-06T01:42:00.046 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.042+0000 7f69a1977700 1 -- 172.21.15.27:0/2909498579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f699c1597c0 msgr2=0x7f699c0cf680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:00.046 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.042+0000 7f69a1977700 1 --2- 172.21.15.27:0/2909498579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f699c1597c0 0x7f699c0cf680 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f699c06a080 tx=0x7f6988009250 comp rx=0 tx=0).stop 2023-10-06T01:42:00.046 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.042+0000 7f69a1977700 1 -- 172.21.15.27:0/2909498579 shutdown_connections 2023-10-06T01:42:00.046 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.042+0000 7f69a1977700 1 --2- 172.21.15.27:0/2909498579 >> v2:172.21.15.124:3300/0 conn(0x7f699c0cfbc0 0x7f699c068350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:00.046 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.042+0000 7f69a1977700 1 --2- 172.21.15.27:0/2909498579 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f699c1593f0 0x7f699c0b00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:00.046 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.042+0000 7f69a1977700 1 --2- 172.21.15.27:0/2909498579 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f699c1597c0 0x7f699c0cf680 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:00.046 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.042+0000 7f69a1977700 1 -- 172.21.15.27:0/2909498579 >> 172.21.15.27:0/2909498579 conn(0x7f699c0d0510 msgr2=0x7f699c0d0910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:00.046 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.042+0000 7f69a1977700 1 -- 172.21.15.27:0/2909498579 shutdown_connections 2023-10-06T01:42:00.046 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.042+0000 7f69a1977700 1 -- 172.21.15.27:0/2909498579 wait complete. 2023-10-06T01:42:00.046 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.043+0000 7f69a1977700 1 Processor -- start 2023-10-06T01:42:00.047 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.044+0000 7f69a1977700 1 -- start start 2023-10-06T01:42:00.047 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.044+0000 7f69a1977700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f699c0cfbc0 0x7f699c155d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:00.047 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.044+0000 7f69a1977700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f699c1593f0 0x7f699c0b1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:00.047 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.044+0000 7f69a1977700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f699c1597c0 0x7f699c0b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:00.047 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.044+0000 7f69a1977700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699c0c8b10 con 0x7f699c0cfbc0 2023-10-06T01:42:00.047 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.044+0000 7f69a1977700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699c1565f0 con 0x7f699c1597c0 2023-10-06T01:42:00.047 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.044+0000 7f69a1977700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f699c1568d0 con 0x7f699c1593f0 2023-10-06T01:42:00.047 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.044+0000 7f69a0975700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f699c0cfbc0 0x7f699c155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:00.047 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.044+0000 7f69a0975700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f699c0cfbc0 0x7f699c155d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53960/0 (socket says 172.21.15.27:53960) 2023-10-06T01:42:00.047 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.045+0000 7f69a0975700 1 -- 172.21.15.27:0/1980085332 learned_addr learned my addr 172.21.15.27:0/1980085332 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:00.047 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.045+0000 7f699bfff700 1 --2- 172.21.15.27:0/1980085332 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f699c1593f0 0x7f699c0b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:00.047 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.045+0000 7f69a1176700 1 --2- 172.21.15.27:0/1980085332 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f699c1597c0 0x7f699c0b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:00.047 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.045+0000 7f69a0975700 1 -- 172.21.15.27:0/1980085332 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f699c1593f0 msgr2=0x7f699c0b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:00.048 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.045+0000 7f69a0975700 1 --2- 172.21.15.27:0/1980085332 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f699c1593f0 0x7f699c0b1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:00.048 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.045+0000 7f69a0975700 1 -- 172.21.15.27:0/1980085332 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f699c1597c0 msgr2=0x7f699c0b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:00.048 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.045+0000 7f69a0975700 1 --2- 172.21.15.27:0/1980085332 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f699c1597c0 0x7f699c0b19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:00.048 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.045+0000 7f69a0975700 1 -- 172.21.15.27:0/1980085332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69880094f0 con 0x7f699c0cfbc0 2023-10-06T01:42:00.048 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.045+0000 7f69a0975700 1 --2- 172.21.15.27:0/1980085332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f699c0cfbc0 0x7f699c155d10 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f699000ec80 tx=0x7f69900091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:00.048 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.045+0000 7f6998ff9700 1 -- 172.21.15.27:0/1980085332 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f699000c850 con 0x7f699c0cfbc0 2023-10-06T01:42:00.048 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.045+0000 7f6998ff9700 1 -- 172.21.15.27:0/1980085332 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f699000c9d0 con 0x7f699c0cfbc0 2023-10-06T01:42:00.048 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.045+0000 7f69a1977700 1 -- 172.21.15.27:0/1980085332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f699c156bb0 con 0x7f699c0cfbc0 2023-10-06T01:42:00.048 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.045+0000 7f6998ff9700 1 -- 172.21.15.27:0/1980085332 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f699000cd90 con 0x7f699c0cfbc0 2023-10-06T01:42:00.048 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.045+0000 7f69a1977700 1 -- 172.21.15.27:0/1980085332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f699c156250 con 0x7f699c0cfbc0 2023-10-06T01:42:00.049 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.046+0000 7f69a1977700 1 -- 172.21.15.27:0/1980085332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f699c0c88e0 con 0x7f699c0cfbc0 2023-10-06T01:42:00.049 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.047+0000 7f6998ff9700 1 -- 172.21.15.27:0/1980085332 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6990013680 con 0x7f699c0cfbc0 2023-10-06T01:42:00.049 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.047+0000 7f6998ff9700 1 --2- 172.21.15.27:0/1980085332 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6980066f30 0x7f69800693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:00.049 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.047+0000 7f6998ff9700 1 -- 172.21.15.27:0/1980085332 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f69900845d0 con 0x7f699c0cfbc0 2023-10-06T01:42:00.050 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.047+0000 7f699bfff700 1 --2- 172.21.15.27:0/1980085332 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6980066f30 0x7f69800693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:00.050 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.048+0000 7f699bfff700 1 --2- 172.21.15.27:0/1980085332 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6980066f30 0x7f69800693e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f699c06a080 tx=0x7f6988017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:00.051 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.051+0000 7f6998ff9700 1 -- 172.21.15.27:0/1980085332 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f699c0c88e0 con 0x7f699c0cfbc0 2023-10-06T01:42:00.242 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.241+0000 7f69a1977700 1 -- 172.21.15.27:0/1980085332 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f699c0c88e0 con 0x7f699c0cfbc0 2023-10-06T01:42:00.242 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.242+0000 7f6998ff9700 1 -- 172.21.15.27:0/1980085332 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v27) v1 ==== 92+0+10 (secure 0 0 0) 0x7f699c0c88e0 con 0x7f699c0cfbc0 2023-10-06T01:42:00.242 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 8 2023-10-06T01:42:00.244 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.244+0000 7f69a1977700 1 -- 172.21.15.27:0/1980085332 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6980066f30 msgr2=0x7f69800693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:00.245 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.244+0000 7f69a1977700 1 --2- 172.21.15.27:0/1980085332 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6980066f30 0x7f69800693e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f699c06a080 tx=0x7f6988017000 comp rx=0 tx=0).stop 2023-10-06T01:42:00.245 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.244+0000 7f69a1977700 1 -- 172.21.15.27:0/1980085332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f699c0cfbc0 msgr2=0x7f699c155d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:00.245 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.244+0000 7f69a1977700 1 --2- 172.21.15.27:0/1980085332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f699c0cfbc0 0x7f699c155d10 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f699000ec80 tx=0x7f69900091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:00.245 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.244+0000 7f69a1977700 1 -- 172.21.15.27:0/1980085332 shutdown_connections 2023-10-06T01:42:00.245 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.244+0000 7f69a1977700 1 --2- 172.21.15.27:0/1980085332 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6980066f30 0x7f69800693e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:00.245 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.244+0000 7f69a1977700 1 --2- 172.21.15.27:0/1980085332 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f699c1597c0 0x7f699c0b19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:00.245 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.244+0000 7f69a1977700 1 --2- 172.21.15.27:0/1980085332 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f699c0cfbc0 0x7f699c155d10 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:00.245 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.244+0000 7f69a1977700 1 --2- 172.21.15.27:0/1980085332 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f699c1593f0 0x7f699c0b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:00.245 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.244+0000 7f69a1977700 1 -- 172.21.15.27:0/1980085332 >> 172.21.15.27:0/1980085332 conn(0x7f699c0d0510 msgr2=0x7f699c0c6b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:00.245 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.244+0000 7f69a1977700 1 -- 172.21.15.27:0/1980085332 shutdown_connections 2023-10-06T01:42:00.245 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.244+0000 7f69a1977700 1 -- 172.21.15.27:0/1980085332 wait complete. 2023-10-06T01:42:00.256 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-06T01:42:00.378 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.377+0000 7f8e59327700 1 Processor -- start 2023-10-06T01:42:00.378 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.377+0000 7f8e59327700 1 -- start start 2023-10-06T01:42:00.380 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.378+0000 7f8e59327700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8e54157b50 0x7f8e540cd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:00.380 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.378+0000 7f8e59327700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f8e540cdb30 0x7f8e54157f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:00.380 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.378+0000 7f8e59327700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e54159a60 con 0x7f8e54158460 2023-10-06T01:42:00.380 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.378+0000 7f8e59327700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e540cdfb0 con 0x7f8e54157b50 2023-10-06T01:42:00.380 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.378+0000 7f8e59327700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e54158e50 con 0x7f8e540cdb30 2023-10-06T01:42:00.380 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.378+0000 7f8e52ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8e54157b50 0x7f8e540cd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:00.381 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.378+0000 7f8e52ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f8e54157b50 0x7f8e540cd5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53968/0 (socket says 172.21.15.27:53968) 2023-10-06T01:42:00.381 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.378+0000 7f8e52ffd700 1 -- 172.21.15.27:0/2777372899 learned_addr learned my addr 172.21.15.27:0/2777372899 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:00.381 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.379+0000 7f8e527fc700 1 --2- 172.21.15.27:0/2777372899 >> v2:172.21.15.124:3300/0 conn(0x7f8e540cdb30 0x7f8e54157f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:00.381 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.379+0000 7f8e52ffd700 1 -- 172.21.15.27:0/2777372899 >> v2:172.21.15.124:3300/0 conn(0x7f8e540cdb30 msgr2=0x7f8e54157f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:00.381 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.379+0000 7f8e52ffd700 1 --2- 172.21.15.27:0/2777372899 >> v2:172.21.15.124:3300/0 conn(0x7f8e540cdb30 0x7f8e54157f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:00.381 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.379+0000 7f8e52ffd700 1 -- 172.21.15.27:0/2777372899 >> v1:172.21.15.124:6789/0 conn(0x7f8e54158460 legacy=0x7f8e54159ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:00.381 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.379+0000 7f8e52ffd700 1 -- 172.21.15.27:0/2777372899 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e54159130 con 0x7f8e54157b50 2023-10-06T01:42:00.381 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.379+0000 7f8e52ffd700 1 --2- 172.21.15.27:0/2777372899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e54157b50 0x7f8e540cd5f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f8e540b0ae0 tx=0x7f8e4c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fb187e6a1dcf059 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:00.381 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.379+0000 7f8e51ffb700 1 -- 172.21.15.27:0/2777372899 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c00b860 con 0x7f8e54157b50 2023-10-06T01:42:00.381 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.379+0000 7f8e51ffb700 1 -- 172.21.15.27:0/2777372899 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e4c00b9c0 con 0x7f8e54157b50 2023-10-06T01:42:00.382 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.379+0000 7f8e51ffb700 1 -- 172.21.15.27:0/2777372899 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c00bd60 con 0x7f8e54157b50 2023-10-06T01:42:00.382 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.380+0000 7f8e59327700 1 -- 172.21.15.27:0/2777372899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e54157b50 msgr2=0x7f8e540cd5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:00.382 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.380+0000 7f8e59327700 1 --2- 172.21.15.27:0/2777372899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e54157b50 0x7f8e540cd5f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f8e540b0ae0 tx=0x7f8e4c009250 comp rx=0 tx=0).stop 2023-10-06T01:42:00.382 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.380+0000 7f8e59327700 1 -- 172.21.15.27:0/2777372899 shutdown_connections 2023-10-06T01:42:00.382 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.380+0000 7f8e59327700 1 --2- 172.21.15.27:0/2777372899 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e54157b50 0x7f8e540cd5f0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:00.382 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.380+0000 7f8e59327700 1 --2- 172.21.15.27:0/2777372899 >> v2:172.21.15.124:3300/0 conn(0x7f8e540cdb30 0x7f8e54157f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:00.382 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.380+0000 7f8e59327700 1 -- 172.21.15.27:0/2777372899 >> 172.21.15.27:0/2777372899 conn(0x7f8e5406d890 msgr2=0x7f8e5406dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:00.382 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.380+0000 7f8e59327700 1 -- 172.21.15.27:0/2777372899 shutdown_connections 2023-10-06T01:42:00.382 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.380+0000 7f8e59327700 1 -- 172.21.15.27:0/2777372899 wait complete. 2023-10-06T01:42:00.385 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.381+0000 7f8e59327700 1 Processor -- start 2023-10-06T01:42:00.385 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.381+0000 7f8e59327700 1 -- start start 2023-10-06T01:42:00.385 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.382+0000 7f8e59327700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f8e540cdb30 0x7f8e54156580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:00.385 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.382+0000 7f8e59327700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e54157b50 0x7f8e54156ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:00.385 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.382+0000 7f8e59327700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8e54158460 0x7f8e5415aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:00.385 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.382+0000 7f8e59327700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e54069d70 con 0x7f8e54157b50 2023-10-06T01:42:00.385 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.382+0000 7f8e59327700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5406a050 con 0x7f8e54158460 2023-10-06T01:42:00.385 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.382+0000 7f8e59327700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5406a330 con 0x7f8e540cdb30 2023-10-06T01:42:00.386 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.382+0000 7f8e527fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e54157b50 0x7f8e54156ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:00.386 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.382+0000 7f8e527fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e54157b50 0x7f8e54156ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53976/0 (socket says 172.21.15.27:53976) 2023-10-06T01:42:00.386 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.382+0000 7f8e527fc700 1 -- 172.21.15.27:0/4113314280 learned_addr learned my addr 172.21.15.27:0/4113314280 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:00.386 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.382+0000 7f8e52ffd700 1 --2- 172.21.15.27:0/4113314280 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f8e540cdb30 0x7f8e54156580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:00.386 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.382+0000 7f8e537fe700 1 --2- 172.21.15.27:0/4113314280 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8e54158460 0x7f8e5415aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:00.386 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.382+0000 7f8e527fc700 1 -- 172.21.15.27:0/4113314280 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f8e540cdb30 msgr2=0x7f8e54156580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:00.386 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.382+0000 7f8e527fc700 1 --2- 172.21.15.27:0/4113314280 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f8e540cdb30 0x7f8e54156580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:00.386 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.382+0000 7f8e527fc700 1 -- 172.21.15.27:0/4113314280 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8e54158460 msgr2=0x7f8e5415aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:00.386 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.382+0000 7f8e527fc700 1 --2- 172.21.15.27:0/4113314280 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8e54158460 0x7f8e5415aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:00.386 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.382+0000 7f8e527fc700 1 -- 172.21.15.27:0/4113314280 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e4c0094f0 con 0x7f8e54157b50 2023-10-06T01:42:00.386 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.382+0000 7f8e527fc700 1 --2- 172.21.15.27:0/4113314280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e54157b50 0x7f8e54156ac0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c00d890 tx=0x7f8e3c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:00.386 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.382+0000 7f8e4affd700 1 -- 172.21.15.27:0/4113314280 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c0125f0 con 0x7f8e54157b50 2023-10-06T01:42:00.387 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.382+0000 7f8e4affd700 1 -- 172.21.15.27:0/4113314280 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e3c012750 con 0x7f8e54157b50 2023-10-06T01:42:00.387 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.382+0000 7f8e4affd700 1 -- 172.21.15.27:0/4113314280 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c012af0 con 0x7f8e54157b50 2023-10-06T01:42:00.387 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.382+0000 7f8e59327700 1 -- 172.21.15.27:0/4113314280 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e5406a610 con 0x7f8e54157b50 2023-10-06T01:42:00.387 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.383+0000 7f8e59327700 1 -- 172.21.15.27:0/4113314280 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e5406ae10 con 0x7f8e54157b50 2023-10-06T01:42:00.387 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.383+0000 7f8e59327700 1 -- 172.21.15.27:0/4113314280 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e54064050 con 0x7f8e54157b50 2023-10-06T01:42:00.387 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.384+0000 7f8e4affd700 1 -- 172.21.15.27:0/4113314280 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8e3c01e460 con 0x7f8e54157b50 2023-10-06T01:42:00.387 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.384+0000 7f8e4affd700 1 --2- 172.21.15.27:0/4113314280 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8e34066f30 0x7f8e340693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:00.387 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.384+0000 7f8e4affd700 1 -- 172.21.15.27:0/4113314280 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f8e3c0848b0 con 0x7f8e54157b50 2023-10-06T01:42:00.387 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.385+0000 7f8e52ffd700 1 --2- 172.21.15.27:0/4113314280 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8e34066f30 0x7f8e340693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:00.388 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.385+0000 7f8e52ffd700 1 --2- 172.21.15.27:0/4113314280 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8e34066f30 0x7f8e340693e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8e540b0ae0 tx=0x7f8e4c017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:00.388 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.388+0000 7f8e4affd700 1 -- 172.21.15.27:0/4113314280 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8e3c051d10 con 0x7f8e54157b50 2023-10-06T01:42:00.582 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.582+0000 7f8e59327700 1 -- 172.21.15.27:0/4113314280 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8e540afc30 con 0x7f8e54157b50 2023-10-06T01:42:00.583 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.582+0000 7f8e4affd700 1 -- 172.21.15.27:0/4113314280 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8e3c055330 con 0x7f8e54157b50 2023-10-06T01:42:00.583 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2023-10-06T01:42:00.585 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.584+0000 7f8e59327700 1 -- 172.21.15.27:0/4113314280 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8e34066f30 msgr2=0x7f8e340693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:00.585 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.584+0000 7f8e59327700 1 --2- 172.21.15.27:0/4113314280 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8e34066f30 0x7f8e340693e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f8e540b0ae0 tx=0x7f8e4c017000 comp rx=0 tx=0).stop 2023-10-06T01:42:00.585 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.584+0000 7f8e59327700 1 -- 172.21.15.27:0/4113314280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e54157b50 msgr2=0x7f8e54156ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:00.585 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.584+0000 7f8e59327700 1 --2- 172.21.15.27:0/4113314280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e54157b50 0x7f8e54156ac0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c00d890 tx=0x7f8e3c008040 comp rx=0 tx=0).stop 2023-10-06T01:42:00.586 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.584+0000 7f8e59327700 1 -- 172.21.15.27:0/4113314280 shutdown_connections 2023-10-06T01:42:00.586 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.584+0000 7f8e59327700 1 --2- 172.21.15.27:0/4113314280 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8e34066f30 0x7f8e340693e0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:00.586 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.584+0000 7f8e59327700 1 --2- 172.21.15.27:0/4113314280 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f8e540cdb30 0x7f8e54156580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:00.586 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.584+0000 7f8e59327700 1 --2- 172.21.15.27:0/4113314280 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f8e54158460 0x7f8e5415aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:00.586 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.584+0000 7f8e59327700 1 --2- 172.21.15.27:0/4113314280 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e54157b50 0x7f8e54156ac0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:00.586 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.584+0000 7f8e59327700 1 -- 172.21.15.27:0/4113314280 >> 172.21.15.27:0/4113314280 conn(0x7f8e5406d890 msgr2=0x7f8e540ccba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:00.586 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.584+0000 7f8e59327700 1 -- 172.21.15.27:0/4113314280 shutdown_connections 2023-10-06T01:42:00.586 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.584+0000 7f8e59327700 1 -- 172.21.15.27:0/4113314280 wait complete. 2023-10-06T01:42:00.597 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-06T01:42:00.716 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.716+0000 7fe9c5935700 1 Processor -- start 2023-10-06T01:42:00.716 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.716+0000 7fe9c5935700 1 -- start start 2023-10-06T01:42:00.719 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.717+0000 7fe9c5935700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fe9c0157570 0x7fe9c00b4060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:00.719 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.717+0000 7fe9c5935700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe9c015b210 0x7fe9c0158560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:00.719 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.717+0000 7fe9c5935700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9c01580d0 con 0x7fe9c00add70 2023-10-06T01:42:00.719 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.717+0000 7fe9c5935700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9c015ba40 con 0x7fe9c015b210 2023-10-06T01:42:00.719 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.717+0000 7fe9c5935700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9c015bd20 con 0x7fe9c0157570 2023-10-06T01:42:00.719 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.717+0000 7fe9be7fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe9c015b210 0x7fe9c0158560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:00.719 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.717+0000 7fe9be7fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe9c015b210 0x7fe9c0158560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53982/0 (socket says 172.21.15.27:53982) 2023-10-06T01:42:00.719 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.717+0000 7fe9be7fc700 1 -- 172.21.15.27:0/1199535264 learned_addr learned my addr 172.21.15.27:0/1199535264 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:00.719 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.717+0000 7fe9beffd700 1 --2- 172.21.15.27:0/1199535264 >> v2:172.21.15.124:3300/0 conn(0x7fe9c0157570 0x7fe9c00b4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:00.719 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.717+0000 7fe9be7fc700 1 -- 172.21.15.27:0/1199535264 >> v2:172.21.15.124:3300/0 conn(0x7fe9c0157570 msgr2=0x7fe9c00b4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:00.719 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.717+0000 7fe9be7fc700 1 --2- 172.21.15.27:0/1199535264 >> v2:172.21.15.124:3300/0 conn(0x7fe9c0157570 0x7fe9c00b4060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:00.720 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.717+0000 7fe9be7fc700 1 -- 172.21.15.27:0/1199535264 >> v1:172.21.15.124:6789/0 conn(0x7fe9c00add70 legacy=0x7fe9c00ae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:00.720 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.717+0000 7fe9be7fc700 1 -- 172.21.15.27:0/1199535264 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c0157d60 con 0x7fe9c015b210 2023-10-06T01:42:00.720 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.717+0000 7fe9be7fc700 1 --2- 172.21.15.27:0/1199535264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9c015b210 0x7fe9c0158560 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0064960 tx=0x7fe9b4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20f88618b475f1b3 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:00.720 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.717+0000 7fe9bdffb700 1 -- 172.21.15.27:0/1199535264 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b4009e70 con 0x7fe9c015b210 2023-10-06T01:42:00.720 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.718+0000 7fe9bdffb700 1 -- 172.21.15.27:0/1199535264 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9b400b860 con 0x7fe9c015b210 2023-10-06T01:42:00.720 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.718+0000 7fe9bdffb700 1 -- 172.21.15.27:0/1199535264 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b400bc00 con 0x7fe9c015b210 2023-10-06T01:42:00.720 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.718+0000 7fe9c5935700 1 -- 172.21.15.27:0/1199535264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9c015b210 msgr2=0x7fe9c0158560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:00.720 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.718+0000 7fe9c5935700 1 --2- 172.21.15.27:0/1199535264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9c015b210 0x7fe9c0158560 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0064960 tx=0x7fe9b4009250 comp rx=0 tx=0).stop 2023-10-06T01:42:00.720 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.718+0000 7fe9c5935700 1 -- 172.21.15.27:0/1199535264 shutdown_connections 2023-10-06T01:42:00.720 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.718+0000 7fe9c5935700 1 --2- 172.21.15.27:0/1199535264 >> v2:172.21.15.124:3300/0 conn(0x7fe9c0157570 0x7fe9c00b4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:00.720 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.718+0000 7fe9c5935700 1 --2- 172.21.15.27:0/1199535264 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9c015b210 0x7fe9c0158560 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:00.720 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.718+0000 7fe9c5935700 1 -- 172.21.15.27:0/1199535264 >> 172.21.15.27:0/1199535264 conn(0x7fe9c0067630 msgr2=0x7fe9c0067a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:00.720 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.718+0000 7fe9c5935700 1 -- 172.21.15.27:0/1199535264 shutdown_connections 2023-10-06T01:42:00.721 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.718+0000 7fe9c5935700 1 -- 172.21.15.27:0/1199535264 wait complete. 2023-10-06T01:42:00.723 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.719+0000 7fe9c5935700 1 Processor -- start 2023-10-06T01:42:00.723 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.719+0000 7fe9c5935700 1 -- start start 2023-10-06T01:42:00.723 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.720+0000 7fe9c5935700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe9c00add70 0x7fe9c00ca920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:00.723 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.720+0000 7fe9c5935700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe9c0157570 0x7fe9c00c54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:00.723 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.720+0000 7fe9c5935700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9c015b210 0x7fe9c01497c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:00.723 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.720+0000 7fe9c5935700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c00c59f0 con 0x7fe9c015b210 2023-10-06T01:42:00.723 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.720+0000 7fe9c5935700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c00c5c90 con 0x7fe9c00add70 2023-10-06T01:42:00.723 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.720+0000 7fe9c5935700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c00c5f70 con 0x7fe9c0157570 2023-10-06T01:42:00.723 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.720+0000 7fe9bf7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9c015b210 0x7fe9c01497c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:00.723 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.720+0000 7fe9bf7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9c015b210 0x7fe9c01497c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53988/0 (socket says 172.21.15.27:53988) 2023-10-06T01:42:00.724 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.720+0000 7fe9bf7fe700 1 -- 172.21.15.27:0/654863133 learned_addr learned my addr 172.21.15.27:0/654863133 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:00.724 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.720+0000 7fe9beffd700 1 --2- 172.21.15.27:0/654863133 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe9c00add70 0x7fe9c00ca920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:00.724 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.720+0000 7fe9be7fc700 1 --2- 172.21.15.27:0/654863133 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe9c0157570 0x7fe9c00c54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:00.724 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.720+0000 7fe9bf7fe700 1 -- 172.21.15.27:0/654863133 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe9c0157570 msgr2=0x7fe9c00c54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:00.724 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.720+0000 7fe9bf7fe700 1 --2- 172.21.15.27:0/654863133 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe9c0157570 0x7fe9c00c54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:00.724 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.721+0000 7fe9bf7fe700 1 -- 172.21.15.27:0/654863133 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe9c00add70 msgr2=0x7fe9c00ca920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:00.724 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.721+0000 7fe9bf7fe700 1 --2- 172.21.15.27:0/654863133 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe9c00add70 0x7fe9c00ca920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:00.724 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.721+0000 7fe9bf7fe700 1 -- 172.21.15.27:0/654863133 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b40094f0 con 0x7fe9c015b210 2023-10-06T01:42:00.724 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.721+0000 7fe9bf7fe700 1 --2- 172.21.15.27:0/654863133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9c015b210 0x7fe9c01497c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe9b800c560 tx=0x7fe9b8007ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:00.724 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.721+0000 7fe9a6ffd700 1 -- 172.21.15.27:0/654863133 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b8011590 con 0x7fe9c015b210 2023-10-06T01:42:00.724 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.721+0000 7fe9a6ffd700 1 -- 172.21.15.27:0/654863133 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9b80116f0 con 0x7fe9c015b210 2023-10-06T01:42:00.724 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.721+0000 7fe9c5935700 1 -- 172.21.15.27:0/654863133 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9c00c6250 con 0x7fe9c015b210 2023-10-06T01:42:00.724 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.721+0000 7fe9a6ffd700 1 -- 172.21.15.27:0/654863133 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9b8011ab0 con 0x7fe9c015b210 2023-10-06T01:42:00.725 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.721+0000 7fe9c5935700 1 -- 172.21.15.27:0/654863133 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9c0149d00 con 0x7fe9c015b210 2023-10-06T01:42:00.725 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.722+0000 7fe9c5935700 1 -- 172.21.15.27:0/654863133 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9c0064050 con 0x7fe9c015b210 2023-10-06T01:42:00.725 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.722+0000 7fe9a6ffd700 1 -- 172.21.15.27:0/654863133 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe9b801b3b0 con 0x7fe9c015b210 2023-10-06T01:42:00.725 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.723+0000 7fe9a6ffd700 1 --2- 172.21.15.27:0/654863133 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe9ac066f30 0x7fe9ac0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:00.725 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.723+0000 7fe9a6ffd700 1 -- 172.21.15.27:0/654863133 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fe9b8083700 con 0x7fe9c015b210 2023-10-06T01:42:00.725 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.723+0000 7fe9beffd700 1 --2- 172.21.15.27:0/654863133 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe9ac066f30 0x7fe9ac0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:00.725 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.723+0000 7fe9beffd700 1 --2- 172.21.15.27:0/654863133 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe9ac066f30 0x7fe9ac0693e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe9b000f570 tx=0x7fe9b000b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:00.727 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.727+0000 7fe9a6ffd700 1 -- 172.21.15.27:0/654863133 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe9b8050b60 con 0x7fe9c015b210 2023-10-06T01:42:00.925 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.925+0000 7fe9c5935700 1 -- 172.21.15.27:0/654863133 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe9c00cdf50 con 0x7fe9c015b210 2023-10-06T01:42:00.926 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.925+0000 7fe9a6ffd700 1 -- 172.21.15.27:0/654863133 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v27) v1 ==== 84+0+126 (secure 0 0 0) 0x7fe9b8017020 con 0x7fe9c015b210 2023-10-06T01:42:00.926 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:00.926 INFO:teuthology.orchestra.run.smithi027.stdout:[ 2023-10-06T01:42:00.926 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:42:00.926 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 1, 2023-10-06T01:42:00.926 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": "rbd" 2023-10-06T01:42:00.926 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2023-10-06T01:42:00.927 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:42:00.927 INFO:teuthology.orchestra.run.smithi027.stdout: "poolnum": 2, 2023-10-06T01:42:00.927 INFO:teuthology.orchestra.run.smithi027.stdout: "poolname": ".mgr" 2023-10-06T01:42:00.927 INFO:teuthology.orchestra.run.smithi027.stdout: } 2023-10-06T01:42:00.927 INFO:teuthology.orchestra.run.smithi027.stdout:] 2023-10-06T01:42:00.928 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.927+0000 7fe9c5935700 1 -- 172.21.15.27:0/654863133 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe9ac066f30 msgr2=0x7fe9ac0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:00.928 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.927+0000 7fe9c5935700 1 --2- 172.21.15.27:0/654863133 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe9ac066f30 0x7fe9ac0693e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe9b000f570 tx=0x7fe9b000b040 comp rx=0 tx=0).stop 2023-10-06T01:42:00.928 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.927+0000 7fe9c5935700 1 -- 172.21.15.27:0/654863133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9c015b210 msgr2=0x7fe9c01497c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:00.928 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.927+0000 7fe9c5935700 1 --2- 172.21.15.27:0/654863133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9c015b210 0x7fe9c01497c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fe9b800c560 tx=0x7fe9b8007ef0 comp rx=0 tx=0).stop 2023-10-06T01:42:00.928 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.927+0000 7fe9c5935700 1 -- 172.21.15.27:0/654863133 shutdown_connections 2023-10-06T01:42:00.928 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.927+0000 7fe9c5935700 1 --2- 172.21.15.27:0/654863133 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe9ac066f30 0x7fe9ac0693e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:00.928 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.927+0000 7fe9c5935700 1 --2- 172.21.15.27:0/654863133 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe9c015b210 0x7fe9c01497c0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:00.928 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.927+0000 7fe9c5935700 1 --2- 172.21.15.27:0/654863133 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe9c00add70 0x7fe9c00ca920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:00.928 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.927+0000 7fe9c5935700 1 --2- 172.21.15.27:0/654863133 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe9c0157570 0x7fe9c00c54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:00.929 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.927+0000 7fe9c5935700 1 -- 172.21.15.27:0/654863133 >> 172.21.15.27:0/654863133 conn(0x7fe9c0067630 msgr2=0x7fe9c015aa80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:00.929 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.927+0000 7fe9c5935700 1 -- 172.21.15.27:0/654863133 shutdown_connections 2023-10-06T01:42:00.929 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:00.927+0000 7fe9c5935700 1 -- 172.21.15.27:0/654863133 wait complete. 2023-10-06T01:42:00.940 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-06T01:42:00.940 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-06T01:42:01.062 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.062+0000 7f937dad0700 1 Processor -- start 2023-10-06T01:42:01.063 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.062+0000 7f937dad0700 1 -- start start 2023-10-06T01:42:01.065 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.063+0000 7f937dad0700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93780cc850 0x7f93780cbb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:01.065 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.063+0000 7f937dad0700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93780cb670 con 0x7f93780cf810 2023-10-06T01:42:01.065 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.063+0000 7f937dad0700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93780cce90 con 0x7f93780cf440 2023-10-06T01:42:01.065 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.063+0000 7f937dad0700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93780cccb0 con 0x7f93780cc850 2023-10-06T01:42:01.065 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.063+0000 7f9377fff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f93780cf810 0x7f9378158e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54430/0 (socket says 172.21.15.27:54430) 2023-10-06T01:42:01.065 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.063+0000 7f9377fff700 1 -- 172.21.15.27:0/2924980696 learned_addr learned my addr 172.21.15.27:0/2924980696 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:01.065 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.063+0000 7f937d2cf700 1 --2- 172.21.15.27:0/2924980696 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93780cc850 0x7f93780cbb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:01.065 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.063+0000 7f93777fe700 1 -- 172.21.15.27:0/2924980696 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2186813998 0 0) 0x7f93780cb670 con 0x7f93780cf810 2023-10-06T01:42:01.066 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.063+0000 7f93777fe700 1 -- 172.21.15.27:0/2924980696 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9360003570 con 0x7f93780cf810 2023-10-06T01:42:01.066 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.064+0000 7f93777fe700 1 -- 172.21.15.27:0/2924980696 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 118401446 0 0) 0x7f93780cce90 con 0x7f93780cf440 2023-10-06T01:42:01.066 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.064+0000 7f93777fe700 1 -- 172.21.15.27:0/2924980696 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93780cb670 con 0x7f93780cf440 2023-10-06T01:42:01.066 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.064+0000 7f937d2cf700 1 -- 172.21.15.27:0/2924980696 >> v1:172.21.15.124:6789/0 conn(0x7f93780cf440 legacy=0x7f93780cc140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:01.066 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.064+0000 7f937d2cf700 1 -- 172.21.15.27:0/2924980696 >> v1:172.21.15.27:6789/0 conn(0x7f93780cf810 legacy=0x7f9378158e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:01.066 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.064+0000 7f937d2cf700 1 -- 172.21.15.27:0/2924980696 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93780cad80 con 0x7f93780cc850 2023-10-06T01:42:01.066 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.064+0000 7f937d2cf700 1 --2- 172.21.15.27:0/2924980696 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93780cc850 0x7f93780cbb00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9368009910 tx=0x7f93680091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=985343bea3c18c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:01.066 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.064+0000 7f93777fe700 1 -- 172.21.15.27:0/2924980696 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9368011630 con 0x7f93780cc850 2023-10-06T01:42:01.066 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.064+0000 7f93777fe700 1 -- 172.21.15.27:0/2924980696 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9368011790 con 0x7f93780cc850 2023-10-06T01:42:01.066 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.064+0000 7f93777fe700 1 -- 172.21.15.27:0/2924980696 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9368011b10 con 0x7f93780cc850 2023-10-06T01:42:01.066 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.064+0000 7f937dad0700 1 -- 172.21.15.27:0/2924980696 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93780cc850 msgr2=0x7f93780cbb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:01.067 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.064+0000 7f937dad0700 1 --2- 172.21.15.27:0/2924980696 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93780cc850 0x7f93780cbb00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9368009910 tx=0x7f93680091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:01.067 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.065+0000 7f937dad0700 1 -- 172.21.15.27:0/2924980696 shutdown_connections 2023-10-06T01:42:01.067 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.065+0000 7f937dad0700 1 --2- 172.21.15.27:0/2924980696 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93780cc850 0x7f93780cbb00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:01.067 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.065+0000 7f937dad0700 1 -- 172.21.15.27:0/2924980696 >> 172.21.15.27:0/2924980696 conn(0x7f93780cd080 msgr2=0x7f93780cd480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:01.067 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.065+0000 7f937dad0700 1 -- 172.21.15.27:0/2924980696 shutdown_connections 2023-10-06T01:42:01.067 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.065+0000 7f937dad0700 1 -- 172.21.15.27:0/2924980696 wait complete. 2023-10-06T01:42:01.069 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.066+0000 7f937dad0700 1 Processor -- start 2023-10-06T01:42:01.069 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.066+0000 7f937dad0700 1 -- start start 2023-10-06T01:42:01.069 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.066+0000 7f937dad0700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93780cc850 0x7f93780b3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:01.070 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.066+0000 7f937dad0700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93780cf440 0x7f937814b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:01.070 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.067+0000 7f937dad0700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f93780cf810 0x7f93780c76c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:01.070 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.067+0000 7f937dad0700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93780c7c00 con 0x7f93780cc850 2023-10-06T01:42:01.070 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.067+0000 7f937dad0700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93780b4320 con 0x7f93780cf810 2023-10-06T01:42:01.070 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.067+0000 7f937dad0700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93780b4600 con 0x7f93780cf440 2023-10-06T01:42:01.070 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.067+0000 7f937cace700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93780cc850 0x7f93780b3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:01.070 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.067+0000 7f937cace700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93780cc850 0x7f93780b3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:53992/0 (socket says 172.21.15.27:53992) 2023-10-06T01:42:01.070 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.067+0000 7f937cace700 1 -- 172.21.15.27:0/317975058 learned_addr learned my addr 172.21.15.27:0/317975058 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:01.070 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.067+0000 7f937d2cf700 1 --2- 172.21.15.27:0/317975058 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f93780cf810 0x7f93780c76c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:01.070 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.067+0000 7f9377fff700 1 --2- 172.21.15.27:0/317975058 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93780cf440 0x7f937814b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:01.070 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.067+0000 7f937cace700 1 -- 172.21.15.27:0/317975058 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93780cf440 msgr2=0x7f937814b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:01.070 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.067+0000 7f937cace700 1 --2- 172.21.15.27:0/317975058 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93780cf440 0x7f937814b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:01.071 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.067+0000 7f937cace700 1 -- 172.21.15.27:0/317975058 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f93780cf810 msgr2=0x7f93780c76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:01.071 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.067+0000 7f937cace700 1 --2- 172.21.15.27:0/317975058 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f93780cf810 0x7f93780c76c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:01.071 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.067+0000 7f937cace700 1 -- 172.21.15.27:0/317975058 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9368009420 con 0x7f93780cc850 2023-10-06T01:42:01.071 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.067+0000 7f937cace700 1 --2- 172.21.15.27:0/317975058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93780cc850 0x7f93780b3d40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f937000cb40 tx=0x7f93700027a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:01.071 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.067+0000 7f9374ff9700 1 -- 172.21.15.27:0/317975058 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93700040f0 con 0x7f93780cc850 2023-10-06T01:42:01.071 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.067+0000 7f9374ff9700 1 -- 172.21.15.27:0/317975058 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f937000a7e0 con 0x7f93780cc850 2023-10-06T01:42:01.071 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.067+0000 7f937dad0700 1 -- 172.21.15.27:0/317975058 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93780b48e0 con 0x7f93780cc850 2023-10-06T01:42:01.071 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.067+0000 7f9374ff9700 1 -- 172.21.15.27:0/317975058 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f937000aba0 con 0x7f93780cc850 2023-10-06T01:42:01.071 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.067+0000 7f937dad0700 1 -- 172.21.15.27:0/317975058 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93780b4ea0 con 0x7f93780cc850 2023-10-06T01:42:01.071 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.068+0000 7f937dad0700 1 -- 172.21.15.27:0/317975058 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93780b50c0 con 0x7f93780cc850 2023-10-06T01:42:01.071 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.069+0000 7f9374ff9700 1 -- 172.21.15.27:0/317975058 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9370012420 con 0x7f93780cc850 2023-10-06T01:42:01.071 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.069+0000 7f9374ff9700 1 --2- 172.21.15.27:0/317975058 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9364066f30 0x7f93640693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:01.072 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.069+0000 7f9374ff9700 1 -- 172.21.15.27:0/317975058 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f9370084320 con 0x7f93780cc850 2023-10-06T01:42:01.072 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.069+0000 7f9377fff700 1 --2- 172.21.15.27:0/317975058 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9364066f30 0x7f93640693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:01.072 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.070+0000 7f9377fff700 1 --2- 172.21.15.27:0/317975058 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9364066f30 0x7f93640693e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9378157280 tx=0x7f936c009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:01.073 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.073+0000 7f9374ff9700 1 -- 172.21.15.27:0/317975058 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9370051840 con 0x7f93780cc850 2023-10-06T01:42:01.265 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.265+0000 7f937dad0700 1 -- 172.21.15.27:0/317975058 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f9378064050 con 0x7f93780cc850 2023-10-06T01:42:01.627 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.627+0000 7f9374ff9700 1 -- 172.21.15.27:0/317975058 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v28) v1 ==== 138+0+0 (secure 0 0 0) 0x7f9370054e60 con 0x7f93780cc850 2023-10-06T01:42:01.630 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.630+0000 7f937dad0700 1 -- 172.21.15.27:0/317975058 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9364066f30 msgr2=0x7f93640693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:01.631 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.630+0000 7f937dad0700 1 --2- 172.21.15.27:0/317975058 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9364066f30 0x7f93640693e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f9378157280 tx=0x7f936c009040 comp rx=0 tx=0).stop 2023-10-06T01:42:01.631 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.630+0000 7f937dad0700 1 -- 172.21.15.27:0/317975058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93780cc850 msgr2=0x7f93780b3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:01.631 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.630+0000 7f937dad0700 1 --2- 172.21.15.27:0/317975058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93780cc850 0x7f93780b3d40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f937000cb40 tx=0x7f93700027a0 comp rx=0 tx=0).stop 2023-10-06T01:42:01.631 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.630+0000 7f937dad0700 1 -- 172.21.15.27:0/317975058 shutdown_connections 2023-10-06T01:42:01.631 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.630+0000 7f937dad0700 1 --2- 172.21.15.27:0/317975058 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9364066f30 0x7f93640693e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:01.631 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.630+0000 7f937dad0700 1 --2- 172.21.15.27:0/317975058 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f93780cf810 0x7f93780c76c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:01.631 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.630+0000 7f937dad0700 1 --2- 172.21.15.27:0/317975058 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93780cc850 0x7f93780b3d40 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:01.631 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.630+0000 7f937dad0700 1 --2- 172.21.15.27:0/317975058 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93780cf440 0x7f937814b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:01.631 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.630+0000 7f937dad0700 1 -- 172.21.15.27:0/317975058 >> 172.21.15.27:0/317975058 conn(0x7f93780cd080 msgr2=0x7f93780b0320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:01.631 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.630+0000 7f937dad0700 1 -- 172.21.15.27:0/317975058 shutdown_connections 2023-10-06T01:42:01.631 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.631+0000 7f937dad0700 1 -- 172.21.15.27:0/317975058 wait complete. 2023-10-06T01:42:01.632 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' created 2023-10-06T01:42:01.645 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-06T01:42:01.771 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.770+0000 7fa88c906700 1 Processor -- start 2023-10-06T01:42:01.771 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.771+0000 7fa88c906700 1 -- start start 2023-10-06T01:42:01.783 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.771+0000 7fa88c906700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa8880b0eb0 0x7fa8880cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:01.783 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.771+0000 7fa88c906700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8880b12b0 con 0x7fa888159890 2023-10-06T01:42:01.783 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.772+0000 7fa88c906700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8880cd860 con 0x7fa8880b1be0 2023-10-06T01:42:01.783 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.772+0000 7fa88c906700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8880ccc50 con 0x7fa8880b0eb0 2023-10-06T01:42:01.783 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.772+0000 7fa88659c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa888159890 0x7fa8880b14d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54444/0 (socket says 172.21.15.27:54444) 2023-10-06T01:42:01.784 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.772+0000 7fa88659c700 1 -- 172.21.15.27:0/954458152 learned_addr learned my addr 172.21.15.27:0/954458152 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:01.784 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.772+0000 7fa886d9d700 1 --2- 172.21.15.27:0/954458152 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa8880b0eb0 0x7fa8880cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:01.784 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.772+0000 7fa88559a700 1 -- 172.21.15.27:0/954458152 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3824784495 0 0) 0x7fa880001290 con 0x7fa888159890 2023-10-06T01:42:01.784 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.772+0000 7fa88559a700 1 -- 172.21.15.27:0/954458152 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa874003570 con 0x7fa888159890 2023-10-06T01:42:01.785 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.772+0000 7fa88559a700 1 -- 172.21.15.27:0/954458152 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1990399521 0 0) 0x7fa874003570 con 0x7fa888159890 2023-10-06T01:42:01.785 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.772+0000 7fa88559a700 1 -- 172.21.15.27:0/954458152 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa8880b0eb0 msgr2=0x7fa8880cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:01.785 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.772+0000 7fa88559a700 1 --2- 172.21.15.27:0/954458152 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa8880b0eb0 0x7fa8880cdcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:01.785 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.772+0000 7fa88559a700 1 -- 172.21.15.27:0/954458152 >> v1:172.21.15.124:6789/0 conn(0x7fa8880b1be0 legacy=0x7fa888159c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:01.785 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.772+0000 7fa88559a700 1 -- 172.21.15.27:0/954458152 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8880ccf30 con 0x7fa888159890 2023-10-06T01:42:01.786 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.772+0000 7fa88559a700 1 -- 172.21.15.27:0/954458152 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2619078944 0 0) 0x7fa880001780 con 0x7fa888159890 2023-10-06T01:42:01.786 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.772+0000 7fa88559a700 1 -- 172.21.15.27:0/954458152 >> v1:172.21.15.27:6789/0 conn(0x7fa888159890 legacy=0x7fa8880b14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:01.786 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.772+0000 7fa88559a700 1 --2- 172.21.15.27:0/954458152 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa8880b0eb0 0x7fa874003a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:01.786 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.772+0000 7fa88559a700 1 --2- 172.21.15.27:0/954458152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa874003fd0 0x7fa874004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:01.786 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.772+0000 7fa88559a700 1 --2- 172.21.15.27:0/954458152 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa874004d50 0x7fa8740051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:01.787 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.772+0000 7fa88559a700 1 -- 172.21.15.27:0/954458152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8880ccc50 con 0x7fa874003fd0 2023-10-06T01:42:01.787 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.772+0000 7fa88559a700 1 -- 172.21.15.27:0/954458152 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87400bb40 con 0x7fa874004d50 2023-10-06T01:42:01.787 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.772+0000 7fa88559a700 1 -- 172.21.15.27:0/954458152 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa87400bd00 con 0x7fa8880b0eb0 2023-10-06T01:42:01.787 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.772+0000 7fa885d9b700 1 --2- 172.21.15.27:0/954458152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa874003fd0 0x7fa874004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:01.787 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.773+0000 7fa88659c700 1 --2- 172.21.15.27:0/954458152 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa8880b0eb0 0x7fa874003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:01.787 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.773+0000 7fa886d9d700 1 --2- 172.21.15.27:0/954458152 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa874004d50 0x7fa8740051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:01.788 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.773+0000 7fa885d9b700 1 -- 172.21.15.27:0/954458152 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa8880b0eb0 msgr2=0x7fa874003a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:01.788 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.773+0000 7fa885d9b700 1 --2- 172.21.15.27:0/954458152 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa8880b0eb0 0x7fa874003a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:01.788 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.773+0000 7fa885d9b700 1 -- 172.21.15.27:0/954458152 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa874004d50 msgr2=0x7fa8740051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:01.788 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.773+0000 7fa885d9b700 1 --2- 172.21.15.27:0/954458152 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa874004d50 0x7fa8740051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:01.788 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.773+0000 7fa885d9b700 1 -- 172.21.15.27:0/954458152 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa874003570 con 0x7fa874003fd0 2023-10-06T01:42:01.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.773+0000 7fa885d9b700 1 --2- 172.21.15.27:0/954458152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa874003fd0 0x7fa874004810 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fa870001970 tx=0x7fa870007d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbd4d4b1f460036f server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:01.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.773+0000 7fa88559a700 1 -- 172.21.15.27:0/954458152 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa87000c5d0 con 0x7fa874003fd0 2023-10-06T01:42:01.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.773+0000 7fa88559a700 1 -- 172.21.15.27:0/954458152 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa87000c750 con 0x7fa874003fd0 2023-10-06T01:42:01.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.773+0000 7fa88559a700 1 -- 172.21.15.27:0/954458152 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa87000cb10 con 0x7fa874003fd0 2023-10-06T01:42:01.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.774+0000 7fa88c906700 1 -- 172.21.15.27:0/954458152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa874003fd0 msgr2=0x7fa874004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:01.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.774+0000 7fa88c906700 1 --2- 172.21.15.27:0/954458152 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa874003fd0 0x7fa874004810 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fa870001970 tx=0x7fa870007d10 comp rx=0 tx=0).stop 2023-10-06T01:42:01.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.774+0000 7fa886d9d700 1 -- 172.21.15.27:0/954458152 reap_dead start 2023-10-06T01:42:01.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.774+0000 7fa88c906700 1 -- 172.21.15.27:0/954458152 shutdown_connections 2023-10-06T01:42:01.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.774+0000 7fa88c906700 1 -- 172.21.15.27:0/954458152 >> 172.21.15.27:0/954458152 conn(0x7fa888099990 msgr2=0x7fa8880b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:01.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.774+0000 7fa88c906700 1 -- 172.21.15.27:0/954458152 shutdown_connections 2023-10-06T01:42:01.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.774+0000 7fa88c906700 1 -- 172.21.15.27:0/954458152 wait complete. 2023-10-06T01:42:01.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.775+0000 7fa88c906700 1 Processor -- start 2023-10-06T01:42:01.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.775+0000 7fa88c906700 1 -- start start 2023-10-06T01:42:01.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.776+0000 7fa88c906700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa874003fd0 0x7fa88815deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:01.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.776+0000 7fa88c906700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa88815e3f0 0x7fa888068ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:01.792 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.776+0000 7fa88c906700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa888069010 0x7fa888069460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:01.792 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.776+0000 7fa88c906700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8880699a0 con 0x7fa874003fd0 2023-10-06T01:42:01.792 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.776+0000 7fa88c906700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa88815c390 con 0x7fa888069010 2023-10-06T01:42:01.792 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.776+0000 7fa88c906700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa88815e860 con 0x7fa88815e3f0 2023-10-06T01:42:01.792 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.776+0000 7fa88659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa874003fd0 0x7fa88815deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:01.793 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.776+0000 7fa88659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa874003fd0 0x7fa88815deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54008/0 (socket says 172.21.15.27:54008) 2023-10-06T01:42:01.793 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.776+0000 7fa88659c700 1 -- 172.21.15.27:0/3667967947 learned_addr learned my addr 172.21.15.27:0/3667967947 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:01.793 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.776+0000 7fa886d9d700 1 --2- 172.21.15.27:0/3667967947 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa888069010 0x7fa888069460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:01.793 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.776+0000 7fa885d9b700 1 --2- 172.21.15.27:0/3667967947 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa88815e3f0 0x7fa888068ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:01.793 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.776+0000 7fa88659c700 1 -- 172.21.15.27:0/3667967947 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa88815e3f0 msgr2=0x7fa888068ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:01.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.776+0000 7fa88659c700 1 --2- 172.21.15.27:0/3667967947 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa88815e3f0 0x7fa888068ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:01.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.776+0000 7fa88659c700 1 -- 172.21.15.27:0/3667967947 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa888069010 msgr2=0x7fa888069460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:01.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.776+0000 7fa88659c700 1 --2- 172.21.15.27:0/3667967947 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa888069010 0x7fa888069460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:01.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.776+0000 7fa88659c700 1 -- 172.21.15.27:0/3667967947 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa88815eb30 con 0x7fa874003fd0 2023-10-06T01:42:01.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.776+0000 7fa885d9b700 1 --2- 172.21.15.27:0/3667967947 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa88815e3f0 0x7fa888068ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:42:01.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.776+0000 7fa886d9d700 1 --2- 172.21.15.27:0/3667967947 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa888069010 0x7fa888069460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:42:01.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.776+0000 7fa88659c700 1 --2- 172.21.15.27:0/3667967947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa874003fd0 0x7fa88815deb0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fa880007370 tx=0x7fa880007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:01.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.776+0000 7fa87e7fc700 1 -- 172.21.15.27:0/3667967947 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8800077e0 con 0x7fa874003fd0 2023-10-06T01:42:01.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.776+0000 7fa87e7fc700 1 -- 172.21.15.27:0/3667967947 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa88000f870 con 0x7fa874003fd0 2023-10-06T01:42:01.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.776+0000 7fa88c906700 1 -- 172.21.15.27:0/3667967947 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa88815ee10 con 0x7fa874003fd0 2023-10-06T01:42:01.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.776+0000 7fa87e7fc700 1 -- 172.21.15.27:0/3667967947 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88000fc30 con 0x7fa874003fd0 2023-10-06T01:42:01.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.776+0000 7fa88c906700 1 -- 172.21.15.27:0/3667967947 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa88815c020 con 0x7fa874003fd0 2023-10-06T01:42:01.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.777+0000 7fa88c906700 1 -- 172.21.15.27:0/3667967947 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa888064050 con 0x7fa874003fd0 2023-10-06T01:42:01.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.778+0000 7fa87e7fc700 1 -- 172.21.15.27:0/3667967947 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa88001c400 con 0x7fa874003fd0 2023-10-06T01:42:01.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.779+0000 7fa87e7fc700 1 --2- 172.21.15.27:0/3667967947 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa868066f30 0x7fa8680693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:01.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.779+0000 7fa87e7fc700 1 -- 172.21.15.27:0/3667967947 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5576+0+0 (secure 0 0 0) 0x7fa880084810 con 0x7fa874003fd0 2023-10-06T01:42:01.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.779+0000 7fa885d9b700 1 --2- 172.21.15.27:0/3667967947 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa868066f30 0x7fa8680693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:01.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.779+0000 7fa885d9b700 1 --2- 172.21.15.27:0/3667967947 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa868066f30 0x7fa8680693e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa888157630 tx=0x7fa870015000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:01.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.783+0000 7fa87e7fc700 1 -- 172.21.15.27:0/3667967947 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa880051b00 con 0x7fa874003fd0 2023-10-06T01:42:01.975 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:01.975+0000 7fa88c906700 1 -- 172.21.15.27:0/3667967947 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fa8880b12b0 con 0x7fa874003fd0 2023-10-06T01:42:02.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.634+0000 7fa87e7fc700 1 -- 172.21.15.27:0/3667967947 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v29) v1 ==== 170+0+0 (secure 0 0 0) 0x7fa8880b12b0 con 0x7fa874003fd0 2023-10-06T01:42:02.641 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.640+0000 7fa88c906700 1 -- 172.21.15.27:0/3667967947 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa868066f30 msgr2=0x7fa8680693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:02.641 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.640+0000 7fa88c906700 1 --2- 172.21.15.27:0/3667967947 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa868066f30 0x7fa8680693e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fa888157630 tx=0x7fa870015000 comp rx=0 tx=0).stop 2023-10-06T01:42:02.641 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.640+0000 7fa88c906700 1 -- 172.21.15.27:0/3667967947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa874003fd0 msgr2=0x7fa88815deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:02.642 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.640+0000 7fa88c906700 1 --2- 172.21.15.27:0/3667967947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa874003fd0 0x7fa88815deb0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fa880007370 tx=0x7fa880007a90 comp rx=0 tx=0).stop 2023-10-06T01:42:02.642 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.640+0000 7fa88c906700 1 -- 172.21.15.27:0/3667967947 shutdown_connections 2023-10-06T01:42:02.642 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.640+0000 7fa88c906700 1 --2- 172.21.15.27:0/3667967947 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa868066f30 0x7fa8680693e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:02.642 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.640+0000 7fa88c906700 1 --2- 172.21.15.27:0/3667967947 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa888069010 0x7fa888069460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:02.642 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.640+0000 7fa88c906700 1 --2- 172.21.15.27:0/3667967947 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa874003fd0 0x7fa88815deb0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:02.642 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.640+0000 7fa88c906700 1 --2- 172.21.15.27:0/3667967947 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa88815e3f0 0x7fa888068ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:02.642 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.640+0000 7fa88c906700 1 -- 172.21.15.27:0/3667967947 >> 172.21.15.27:0/3667967947 conn(0x7fa888099990 msgr2=0x7fa888158bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:02.643 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.642+0000 7fa88c906700 1 -- 172.21.15.27:0/3667967947 shutdown_connections 2023-10-06T01:42:02.643 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.643+0000 7fa88c906700 1 -- 172.21.15.27:0/3667967947 wait complete. 2023-10-06T01:42:02.645 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' created 2023-10-06T01:42:02.661 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-06T01:42:02.787 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.786+0000 7f91ed512700 1 Processor -- start 2023-10-06T01:42:02.787 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.786+0000 7f91ed512700 1 -- start start 2023-10-06T01:42:02.788 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.787+0000 7f91ed512700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f91e80c80d0 0x7f91e815d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:02.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.787+0000 7f91ed512700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f91e806f270 0x7f91e8159a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:02.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.787+0000 7f91ed512700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91e80ae340 con 0x7f91e806b470 2023-10-06T01:42:02.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.787+0000 7f91ed512700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91e815e290 con 0x7f91e806f270 2023-10-06T01:42:02.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.787+0000 7f91ed512700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91e80c8870 con 0x7f91e80c80d0 2023-10-06T01:42:02.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.787+0000 7f91e6ffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f91e806b470 0x7f91e80c79c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54460/0 (socket says 172.21.15.27:54460) 2023-10-06T01:42:02.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.787+0000 7f91e6ffd700 1 -- 172.21.15.27:0/3812769954 learned_addr learned my addr 172.21.15.27:0/3812769954 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:02.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.788+0000 7f91e67fc700 1 --2- 172.21.15.27:0/3812769954 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f91e80c80d0 0x7f91e815d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:02.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.788+0000 7f91e77fe700 1 --2- 172.21.15.27:0/3812769954 >> v2:172.21.15.124:3300/0 conn(0x7f91e806f270 0x7f91e8159a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:02.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.788+0000 7f91e5ffb700 1 -- 172.21.15.27:0/3812769954 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3763066467 0 0) 0x7f91e80ae340 con 0x7f91e806b470 2023-10-06T01:42:02.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.788+0000 7f91e5ffb700 1 -- 172.21.15.27:0/3812769954 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91d4003570 con 0x7f91e806b470 2023-10-06T01:42:02.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.788+0000 7f91e67fc700 1 -- 172.21.15.27:0/3812769954 >> v2:172.21.15.124:3300/0 conn(0x7f91e806f270 msgr2=0x7f91e8159a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:02.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.788+0000 7f91e67fc700 1 --2- 172.21.15.27:0/3812769954 >> v2:172.21.15.124:3300/0 conn(0x7f91e806f270 0x7f91e8159a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:02.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.788+0000 7f91e67fc700 1 -- 172.21.15.27:0/3812769954 >> v1:172.21.15.27:6789/0 conn(0x7f91e806b470 legacy=0x7f91e80c79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:02.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.788+0000 7f91e67fc700 1 -- 172.21.15.27:0/3812769954 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e80afaa0 con 0x7f91e80c80d0 2023-10-06T01:42:02.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.788+0000 7f91e77fe700 1 --2- 172.21.15.27:0/3812769954 >> v2:172.21.15.124:3300/0 conn(0x7f91e806f270 0x7f91e8159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:42:02.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.788+0000 7f91e67fc700 1 --2- 172.21.15.27:0/3812769954 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f91e80c80d0 0x7f91e815d2f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f91d00098e0 tx=0x7f91d00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5adc02abd8f82d84 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:02.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.788+0000 7f91e5ffb700 1 -- 172.21.15.27:0/3812769954 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d0011630 con 0x7f91e80c80d0 2023-10-06T01:42:02.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.788+0000 7f91e5ffb700 1 -- 172.21.15.27:0/3812769954 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d0011790 con 0x7f91e80c80d0 2023-10-06T01:42:02.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.788+0000 7f91e5ffb700 1 -- 172.21.15.27:0/3812769954 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d0011b10 con 0x7f91e80c80d0 2023-10-06T01:42:02.792 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.789+0000 7f91ed512700 1 -- 172.21.15.27:0/3812769954 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f91e80c80d0 msgr2=0x7f91e815d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:02.793 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.789+0000 7f91ed512700 1 --2- 172.21.15.27:0/3812769954 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f91e80c80d0 0x7f91e815d2f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f91d00098e0 tx=0x7f91d00091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:02.793 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.789+0000 7f91ed512700 1 -- 172.21.15.27:0/3812769954 shutdown_connections 2023-10-06T01:42:02.793 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.789+0000 7f91ed512700 1 --2- 172.21.15.27:0/3812769954 >> v2:172.21.15.124:3300/0 conn(0x7f91e806f270 0x7f91e8159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:02.793 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.789+0000 7f91ed512700 1 --2- 172.21.15.27:0/3812769954 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f91e80c80d0 0x7f91e815d2f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:02.793 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.789+0000 7f91ed512700 1 -- 172.21.15.27:0/3812769954 >> 172.21.15.27:0/3812769954 conn(0x7f91e806e760 msgr2=0x7f91e806eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:02.793 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.789+0000 7f91ed512700 1 -- 172.21.15.27:0/3812769954 shutdown_connections 2023-10-06T01:42:02.793 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.789+0000 7f91ed512700 1 -- 172.21.15.27:0/3812769954 wait complete. 2023-10-06T01:42:02.793 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.790+0000 7f91ed512700 1 Processor -- start 2023-10-06T01:42:02.793 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.790+0000 7f91ed512700 1 -- start start 2023-10-06T01:42:02.793 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.791+0000 7f91ed512700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91e806b470 0x7f91e8156d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:02.793 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.791+0000 7f91ed512700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f91e806f270 0x7f91e80c68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:02.793 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.791+0000 7f91ed512700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f91e80c80d0 0x7f91e80c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:02.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.791+0000 7f91ed512700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e80b0350 con 0x7f91e806b470 2023-10-06T01:42:02.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.791+0000 7f91ed512700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e80b05a0 con 0x7f91e806f270 2023-10-06T01:42:02.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.791+0000 7f91ed512700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91e80b0880 con 0x7f91e80c80d0 2023-10-06T01:42:02.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.791+0000 7f91e6ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91e806b470 0x7f91e8156d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:02.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.791+0000 7f91e6ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91e806b470 0x7f91e8156d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54024/0 (socket says 172.21.15.27:54024) 2023-10-06T01:42:02.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.791+0000 7f91e6ffd700 1 -- 172.21.15.27:0/4228905345 learned_addr learned my addr 172.21.15.27:0/4228905345 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:02.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.791+0000 7f91e77fe700 1 --2- 172.21.15.27:0/4228905345 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f91e80c80d0 0x7f91e80c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:02.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.791+0000 7f91e67fc700 1 --2- 172.21.15.27:0/4228905345 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f91e806f270 0x7f91e80c68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:02.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.791+0000 7f91e6ffd700 1 -- 172.21.15.27:0/4228905345 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f91e80c80d0 msgr2=0x7f91e80c6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:02.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.791+0000 7f91e6ffd700 1 --2- 172.21.15.27:0/4228905345 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f91e80c80d0 0x7f91e80c6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:02.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.791+0000 7f91e6ffd700 1 -- 172.21.15.27:0/4228905345 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f91e806f270 msgr2=0x7f91e80c68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:02.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.791+0000 7f91e6ffd700 1 --2- 172.21.15.27:0/4228905345 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f91e806f270 0x7f91e80c68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:02.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.792+0000 7f91e6ffd700 1 -- 172.21.15.27:0/4228905345 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e0009a20 con 0x7f91e806b470 2023-10-06T01:42:02.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.792+0000 7f91e6ffd700 1 --2- 172.21.15.27:0/4228905345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91e806b470 0x7f91e8156d80 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f91d80011e0 tx=0x7f91d8007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:02.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.792+0000 7f91deffd700 1 -- 172.21.15.27:0/4228905345 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d8005b30 con 0x7f91e806b470 2023-10-06T01:42:02.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.792+0000 7f91deffd700 1 -- 172.21.15.27:0/4228905345 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d8005c90 con 0x7f91e806b470 2023-10-06T01:42:02.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.792+0000 7f91ed512700 1 -- 172.21.15.27:0/4228905345 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91d0009420 con 0x7f91e806b470 2023-10-06T01:42:02.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.792+0000 7f91deffd700 1 -- 172.21.15.27:0/4228905345 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d8007300 con 0x7f91e806b470 2023-10-06T01:42:02.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.792+0000 7f91ed512700 1 -- 172.21.15.27:0/4228905345 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91e80b0e40 con 0x7f91e806b470 2023-10-06T01:42:02.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.793+0000 7f91ed512700 1 -- 172.21.15.27:0/4228905345 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91e8064050 con 0x7f91e806b470 2023-10-06T01:42:02.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.793+0000 7f91deffd700 1 -- 172.21.15.27:0/4228905345 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f91d80076b0 con 0x7f91e806b470 2023-10-06T01:42:02.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.794+0000 7f91deffd700 1 --2- 172.21.15.27:0/4228905345 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f91c8066f30 0x7f91c80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:02.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.794+0000 7f91deffd700 1 -- 172.21.15.27:0/4228905345 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5958+0+0 (secure 0 0 0) 0x7f91d8083b70 con 0x7f91e806b470 2023-10-06T01:42:02.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.794+0000 7f91e67fc700 1 --2- 172.21.15.27:0/4228905345 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f91c8066f30 0x7f91c80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:02.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.794+0000 7f91e67fc700 1 --2- 172.21.15.27:0/4228905345 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f91c8066f30 0x7f91c80693e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f91d00091e0 tx=0x7f91d0017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:02.798 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:02.798+0000 7f91deffd700 1 -- 172.21.15.27:0/4228905345 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f91d8050c40 con 0x7f91e806b470 2023-10-06T01:42:03.035 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.035+0000 7f91ed512700 1 -- 172.21.15.27:0/4228905345 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f91e80cc310 con 0x7f91e806b470 2023-10-06T01:42:03.635 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.635+0000 7f91deffd700 1 -- 172.21.15.27:0/4228905345 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v12) v1 ==== 172+0+0 (secure 0 0 0) 0x7f91d8017020 con 0x7f91e806b470 2023-10-06T01:42:03.643 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.641+0000 7f91ed512700 1 -- 172.21.15.27:0/4228905345 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f91c8066f30 msgr2=0x7f91c80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:03.643 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.641+0000 7f91ed512700 1 --2- 172.21.15.27:0/4228905345 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f91c8066f30 0x7f91c80693e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f91d00091e0 tx=0x7f91d0017000 comp rx=0 tx=0).stop 2023-10-06T01:42:03.643 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.641+0000 7f91ed512700 1 -- 172.21.15.27:0/4228905345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91e806b470 msgr2=0x7f91e8156d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:03.643 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.641+0000 7f91ed512700 1 --2- 172.21.15.27:0/4228905345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91e806b470 0x7f91e8156d80 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f91d80011e0 tx=0x7f91d8007df0 comp rx=0 tx=0).stop 2023-10-06T01:42:03.643 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.641+0000 7f91ed512700 1 -- 172.21.15.27:0/4228905345 shutdown_connections 2023-10-06T01:42:03.643 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.641+0000 7f91ed512700 1 --2- 172.21.15.27:0/4228905345 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f91c8066f30 0x7f91c80693e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:03.644 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.641+0000 7f91ed512700 1 --2- 172.21.15.27:0/4228905345 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f91e80c80d0 0x7f91e80c6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:03.644 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.641+0000 7f91ed512700 1 --2- 172.21.15.27:0/4228905345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f91e806b470 0x7f91e8156d80 secure :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f91d80011e0 tx=0x7f91d8007df0 comp rx=0 tx=0).stop 2023-10-06T01:42:03.644 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.641+0000 7f91ed512700 1 --2- 172.21.15.27:0/4228905345 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f91e806f270 0x7f91e80c68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:03.644 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.641+0000 7f91ed512700 1 -- 172.21.15.27:0/4228905345 >> 172.21.15.27:0/4228905345 conn(0x7f91e806e760 msgr2=0x7f91e806eb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:03.644 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.641+0000 7f91ed512700 1 -- 172.21.15.27:0/4228905345 shutdown_connections 2023-10-06T01:42:03.644 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.643+0000 7f91ed512700 1 -- 172.21.15.27:0/4228905345 wait complete. 2023-10-06T01:42:03.651 INFO:teuthology.orchestra.run.smithi027.stderr:new fs with metadata pool 5 and data pool 6 2023-10-06T01:42:03.667 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T01:42:03.787 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.787+0000 7f2f0d9a1700 1 Processor -- start 2023-10-06T01:42:03.788 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.787+0000 7f2f0d9a1700 1 -- start start 2023-10-06T01:42:03.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.788+0000 7f2f0d9a1700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2f080cbda0 0x7f2f08067630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:03.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.788+0000 7f2f0d9a1700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2f08067b70 0x7f2f0815c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:03.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.788+0000 7f2f0d9a1700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f2f080cc170 0x7f2f0815cc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:03.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.788+0000 7f2f0d9a1700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f080af040 con 0x7f2f080cbda0 2023-10-06T01:42:03.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.788+0000 7f2f0d9a1700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f080afd70 con 0x7f2f080cc170 2023-10-06T01:42:03.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.788+0000 7f2f0d9a1700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f08068080 con 0x7f2f08067b70 2023-10-06T01:42:03.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.788+0000 7f2f0c99f700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2f080cbda0 0x7f2f08067630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:03.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.788+0000 7f2f0c99f700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2f080cbda0 0x7f2f08067630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54032/0 (socket says 172.21.15.27:54032) 2023-10-06T01:42:03.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.788+0000 7f2f0c99f700 1 -- 172.21.15.27:0/1030485991 learned_addr learned my addr 172.21.15.27:0/1030485991 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:03.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.788+0000 7f2f07fff700 1 --2- 172.21.15.27:0/1030485991 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2f08067b70 0x7f2f0815c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:03.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.788+0000 7f2f0d1a0700 1 --2- 172.21.15.27:0/1030485991 >> v2:172.21.15.124:3300/0 conn(0x7f2f080cc170 0x7f2f0815cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:03.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.788+0000 7f2f0c99f700 1 -- 172.21.15.27:0/1030485991 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2f08067b70 msgr2=0x7f2f0815c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:03.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.788+0000 7f2f0c99f700 1 --2- 172.21.15.27:0/1030485991 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2f08067b70 0x7f2f0815c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:03.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.788+0000 7f2f0c99f700 1 -- 172.21.15.27:0/1030485991 >> v2:172.21.15.124:3300/0 conn(0x7f2f080cc170 msgr2=0x7f2f0815cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:03.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.788+0000 7f2f0c99f700 1 --2- 172.21.15.27:0/1030485991 >> v2:172.21.15.124:3300/0 conn(0x7f2f080cc170 0x7f2f0815cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:03.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.788+0000 7f2f0c99f700 1 -- 172.21.15.27:0/1030485991 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f080af290 con 0x7f2f080cbda0 2023-10-06T01:42:03.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.788+0000 7f2f0d1a0700 1 --2- 172.21.15.27:0/1030485991 >> v2:172.21.15.124:3300/0 conn(0x7f2f080cc170 0x7f2f0815cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:42:03.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.788+0000 7f2f07fff700 1 --2- 172.21.15.27:0/1030485991 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2f08067b70 0x7f2f0815c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:42:03.792 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.789+0000 7f2f0c99f700 1 --2- 172.21.15.27:0/1030485991 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f080cbda0 0x7f2f08067630 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f2f080ccf10 tx=0x7f2f00009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eb3d31be5050a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:03.792 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.789+0000 7f2f077fe700 1 -- 172.21.15.27:0/1030485991 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f0000b860 con 0x7f2f080cbda0 2023-10-06T01:42:03.792 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.789+0000 7f2f077fe700 1 -- 172.21.15.27:0/1030485991 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f0000b9c0 con 0x7f2f080cbda0 2023-10-06T01:42:03.792 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.789+0000 7f2f077fe700 1 -- 172.21.15.27:0/1030485991 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f0000bd60 con 0x7f2f080cbda0 2023-10-06T01:42:03.792 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.789+0000 7f2f0d9a1700 1 -- 172.21.15.27:0/1030485991 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f080cbda0 msgr2=0x7f2f08067630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:03.792 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.789+0000 7f2f0d9a1700 1 --2- 172.21.15.27:0/1030485991 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f080cbda0 0x7f2f08067630 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f2f080ccf10 tx=0x7f2f00009250 comp rx=0 tx=0).stop 2023-10-06T01:42:03.792 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.789+0000 7f2f0d9a1700 1 -- 172.21.15.27:0/1030485991 shutdown_connections 2023-10-06T01:42:03.792 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.789+0000 7f2f0d9a1700 1 --2- 172.21.15.27:0/1030485991 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f080cbda0 0x7f2f08067630 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:03.792 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.789+0000 7f2f0d9a1700 1 --2- 172.21.15.27:0/1030485991 >> v2:172.21.15.124:3300/0 conn(0x7f2f080cc170 0x7f2f0815cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:03.792 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.789+0000 7f2f0d9a1700 1 --2- 172.21.15.27:0/1030485991 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2f08067b70 0x7f2f0815c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:03.792 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.789+0000 7f2f0d9a1700 1 -- 172.21.15.27:0/1030485991 >> 172.21.15.27:0/1030485991 conn(0x7f2f080d0f70 msgr2=0x7f2f080ccf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:03.792 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.789+0000 7f2f0d9a1700 1 -- 172.21.15.27:0/1030485991 shutdown_connections 2023-10-06T01:42:03.793 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.789+0000 7f2f0d9a1700 1 -- 172.21.15.27:0/1030485991 wait complete. 2023-10-06T01:42:03.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.790+0000 7f2f0d9a1700 1 Processor -- start 2023-10-06T01:42:03.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.791+0000 7f2f0d9a1700 1 -- start start 2023-10-06T01:42:03.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.791+0000 7f2f0d9a1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f08067b70 0x7f2f080c75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:03.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.791+0000 7f2f0d9a1700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2f080cbda0 0x7f2f080b4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:03.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.791+0000 7f2f0d9a1700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2f080cc170 0x7f2f0815dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:03.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.791+0000 7f2f0d9a1700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f080c89a0 con 0x7f2f08067b70 2023-10-06T01:42:03.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.791+0000 7f2f0d9a1700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f080b4690 con 0x7f2f080cbda0 2023-10-06T01:42:03.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.791+0000 7f2f0d9a1700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0815e000 con 0x7f2f080cc170 2023-10-06T01:42:03.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.791+0000 7f2f0c99f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f08067b70 0x7f2f080c75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:03.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.791+0000 7f2f0c99f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f08067b70 0x7f2f080c75b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54040/0 (socket says 172.21.15.27:54040) 2023-10-06T01:42:03.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.791+0000 7f2f0c99f700 1 -- 172.21.15.27:0/1435697196 learned_addr learned my addr 172.21.15.27:0/1435697196 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:03.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.792+0000 7f2f07fff700 1 --2- 172.21.15.27:0/1435697196 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2f080cbda0 0x7f2f080b4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:03.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.792+0000 7f2f0d1a0700 1 --2- 172.21.15.27:0/1435697196 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2f080cc170 0x7f2f0815dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:03.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.792+0000 7f2f0c99f700 1 -- 172.21.15.27:0/1435697196 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2f080cc170 msgr2=0x7f2f0815dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:03.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.792+0000 7f2f0c99f700 1 --2- 172.21.15.27:0/1435697196 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2f080cc170 0x7f2f0815dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:03.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.792+0000 7f2f0c99f700 1 -- 172.21.15.27:0/1435697196 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2f080cbda0 msgr2=0x7f2f080b4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:03.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.792+0000 7f2f0c99f700 1 --2- 172.21.15.27:0/1435697196 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2f080cbda0 0x7f2f080b4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:03.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.792+0000 7f2f0c99f700 1 -- 172.21.15.27:0/1435697196 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f000094f0 con 0x7f2f08067b70 2023-10-06T01:42:03.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.792+0000 7f2f0c99f700 1 --2- 172.21.15.27:0/1435697196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f08067b70 0x7f2f080c75b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f2f08089480 tx=0x7f2f00017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:03.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.792+0000 7f2f04ff9700 1 -- 172.21.15.27:0/1435697196 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f00011dc0 con 0x7f2f08067b70 2023-10-06T01:42:03.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.792+0000 7f2f04ff9700 1 -- 172.21.15.27:0/1435697196 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f00024440 con 0x7f2f08067b70 2023-10-06T01:42:03.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.792+0000 7f2f0d9a1700 1 -- 172.21.15.27:0/1435697196 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f080b4080 con 0x7f2f08067b70 2023-10-06T01:42:03.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.792+0000 7f2f04ff9700 1 -- 172.21.15.27:0/1435697196 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f00024800 con 0x7f2f08067b70 2023-10-06T01:42:03.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.792+0000 7f2f0d9a1700 1 -- 172.21.15.27:0/1435697196 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f080b5f30 con 0x7f2f08067b70 2023-10-06T01:42:03.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.793+0000 7f2f0d9a1700 1 -- 172.21.15.27:0/1435697196 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f080b4970 con 0x7f2f08067b70 2023-10-06T01:42:03.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.794+0000 7f2f04ff9700 1 -- 172.21.15.27:0/1435697196 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2f0001a420 con 0x7f2f08067b70 2023-10-06T01:42:03.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.794+0000 7f2f04ff9700 1 --2- 172.21.15.27:0/1435697196 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2ee8066f30 0x7f2ee80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:03.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.794+0000 7f2f04ff9700 1 -- 172.21.15.27:0/1435697196 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f2f000864e0 con 0x7f2f08067b70 2023-10-06T01:42:03.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.794+0000 7f2f07fff700 1 --2- 172.21.15.27:0/1435697196 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2ee8066f30 0x7f2ee80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:03.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.795+0000 7f2f07fff700 1 --2- 172.21.15.27:0/1435697196 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2ee8066f30 0x7f2ee80693e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2f080b62c0 tx=0x7f2ef8007720 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:03.798 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.799+0000 7f2f04ff9700 1 -- 172.21.15.27:0/1435697196 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2f00053670 con 0x7f2f08067b70 2023-10-06T01:42:03.995 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.995+0000 7f2f0d9a1700 1 -- 172.21.15.27:0/1435697196 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2f0804c3c0 con 0x7f2f08067b70 2023-10-06T01:42:03.996 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.996+0000 7f2f04ff9700 1 -- 172.21.15.27:0/1435697196 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+16879 (secure 0 0 0) 0x7f2f00056c90 con 0x7f2f08067b70 2023-10-06T01:42:03.996 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:03.996 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":30,"fsid":"20a7bff0-74f8-4f2f-b6c6-899e804bc965","created":"2023-10-06T01:40:54.474215+0000","modified":"2023-10-06T01:42:03.036481+0000","last_up_change":"2023-10-06T01:41:05.242847+0000","last_in_change":"2023-10-06T01:40:56.890960+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-10-06T01:41:05.685386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-10-06T01:41:06.092995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-10-06T01:42:01.295053+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-10-06T01:42:02.005152+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b902c86d-800c-40c5-b152-b545e50d02c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":90644},{"type":"v1","addr":"172.21.15.27:6801","nonce":90644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":90644},{"type":"v1","addr":"172.21.15.27:6803","nonce":90644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":90644},{"type":"v1","addr":"172.21.15.27:6807","nonce":90644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":90644},{"type":"v1","addr":"172.21.15.27:6805","nonce":90644}]},"public_addr":"172.21.15.27:6801/90644","cluster_addr":"172.21.15.27:6803/90644","heartbeat_back_addr":"172.21.15.27:6807/90644","heartbeat_front_addr":"172.21.15.27:6805/90644","state":["exists","up"]},{"osd":1,"uuid":"5b4a70e5-b06e-48cd-bfc4-7962c5d05386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":90643},{"type":"v1","addr":"172.21.15.27:6809","nonce":90643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":90643},{"type":"v1","addr":"172.21.15.27:6811","nonce":90643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":90643},{"type":"v1","addr":"172.21.15.27:6815","nonce":90643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":90643},{"type":"v1","addr":"172.21.15.27:6813","nonce":90643}]},"public_addr":"172.21.15.27:6809/90643","cluster_addr":"172.21.15.27:6811/90643","heartbeat_back_addr":"172.21.15.27:6815/90643","heartbeat_front_addr":"172.21.15.27:6813/90643","state":["exists","up"]},{"osd":2,"uuid":"22d9de46-4073-4336-9122-5227e601eca0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":90645},{"type":"v1","addr":"172.21.15.27:6817","nonce":90645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":90645},{"type":"v1","addr":"172.21.15.27:6819","nonce":90645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":90645},{"type":"v1","addr":"172.21.15.27:6823","nonce":90645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":90645},{"type":"v1","addr":"172.21.15.27:6821","nonce":90645}]},"public_addr":"172.21.15.27:6817/90645","cluster_addr":"172.21.15.27:6819/90645","heartbeat_back_addr":"172.21.15.27:6823/90645","heartbeat_front_addr":"172.21.15.27:6821/90645","state":["exists","up"]},{"osd":3,"uuid":"438bb2d3-34e2-4b7c-9734-8360c1b67650","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":90668},{"type":"v1","addr":"172.21.15.27:6825","nonce":90668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":90668},{"type":"v1","addr":"172.21.15.27:6827","nonce":90668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":90668},{"type":"v1","addr":"172.21.15.27:6831","nonce":90668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":90668},{"type":"v1","addr":"172.21.15.27:6829","nonce":90668}]},"public_addr":"172.21.15.27:6825/90668","cluster_addr":"172.21.15.27:6827/90668","heartbeat_back_addr":"172.21.15.27:6831/90668","heartbeat_front_addr":"172.21.15.27:6829/90668","state":["exists","up"]},{"osd":4,"uuid":"e1956a8e-e0f8-4e7c-acbb-40037415a354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":90863},{"type":"v1","addr":"172.21.15.124:6801","nonce":90863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":90863},{"type":"v1","addr":"172.21.15.124:6803","nonce":90863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":90863},{"type":"v1","addr":"172.21.15.124:6807","nonce":90863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":90863},{"type":"v1","addr":"172.21.15.124:6805","nonce":90863}]},"public_addr":"172.21.15.124:6801/90863","cluster_addr":"172.21.15.124:6803/90863","heartbeat_back_addr":"172.21.15.124:6807/90863","heartbeat_front_addr":"172.21.15.124:6805/90863","state":["exists","up"]},{"osd":5,"uuid":"306c59d4-846d-4b73-b1c9-051bec9e9368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":90885},{"type":"v1","addr":"172.21.15.124:6817","nonce":90885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":90885},{"type":"v1","addr":"172.21.15.124:6819","nonce":90885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":90885},{"type":"v1","addr":"172.21.15.124:6823","nonce":90885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":90885},{"type":"v1","addr":"172.21.15.124:6821","nonce":90885}]},"public_addr":"172.21.15.124:6817/90885","cluster_addr":"172.21.15.124:6819/90885","heartbeat_back_addr":"172.21.15.124:6823/90885","heartbeat_front_addr":"172.21.15.124:6821/90885","state":["exists","up"]},{"osd":6,"uuid":"15463e53-ac8d-47d1-b4f6-c0d31f29e4a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":90864},{"type":"v1","addr":"172.21.15.124:6809","nonce":90864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":90864},{"type":"v1","addr":"172.21.15.124:6811","nonce":90864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":90864},{"type":"v1","addr":"172.21.15.124:6815","nonce":90864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":90864},{"type":"v1","addr":"172.21.15.124:6813","nonce":90864}]},"public_addr":"172.21.15.124:6809/90864","cluster_addr":"172.21.15.124:6811/90864","heartbeat_back_addr":"172.21.15.124:6815/90864","heartbeat_front_addr":"172.21.15.124:6813/90864","state":["exists","up"]},{"osd":7,"uuid":"ae463a90-a9d3-498f-86fa-47a35b3fe889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":90894},{"type":"v1","addr":"172.21.15.124:6825","nonce":90894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":90894},{"type":"v1","addr":"172.21.15.124:6827","nonce":90894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":90894},{"type":"v1","addr":"172.21.15.124:6831","nonce":90894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":90894},{"type":"v1","addr":"172.21.15.124:6829","nonce":90894}]},"public_addr":"172.21.15.124:6825/90894","cluster_addr":"172.21.15.124:6827/90894","heartbeat_back_addr":"172.21.15.124:6831/90894","heartbeat_front_addr":"172.21.15.124:6829/90894","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.827704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.839951+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.032230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.050605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.987908+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.031290+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.080707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.053692+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T01:42:03.998 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.998+0000 7f2f0d9a1700 1 -- 172.21.15.27:0/1435697196 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2ee8066f30 msgr2=0x7f2ee80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:03.998 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.998+0000 7f2f0d9a1700 1 --2- 172.21.15.27:0/1435697196 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2ee8066f30 0x7f2ee80693e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2f080b62c0 tx=0x7f2ef8007720 comp rx=0 tx=0).stop 2023-10-06T01:42:03.999 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.998+0000 7f2f0d9a1700 1 -- 172.21.15.27:0/1435697196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f08067b70 msgr2=0x7f2f080c75b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:03.999 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.998+0000 7f2f0d9a1700 1 --2- 172.21.15.27:0/1435697196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f08067b70 0x7f2f080c75b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f2f08089480 tx=0x7f2f00017000 comp rx=0 tx=0).stop 2023-10-06T01:42:03.999 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.998+0000 7f2f0d9a1700 1 -- 172.21.15.27:0/1435697196 shutdown_connections 2023-10-06T01:42:03.999 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.998+0000 7f2f0d9a1700 1 --2- 172.21.15.27:0/1435697196 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2ee8066f30 0x7f2ee80693e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:03.999 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.998+0000 7f2f0d9a1700 1 --2- 172.21.15.27:0/1435697196 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2f080cc170 0x7f2f0815dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:03.999 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.998+0000 7f2f0d9a1700 1 --2- 172.21.15.27:0/1435697196 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f08067b70 0x7f2f080c75b0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:03.999 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.998+0000 7f2f0d9a1700 1 --2- 172.21.15.27:0/1435697196 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2f080cbda0 0x7f2f080b4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:03.999 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.998+0000 7f2f0d9a1700 1 -- 172.21.15.27:0/1435697196 >> 172.21.15.27:0/1435697196 conn(0x7f2f080d0f70 msgr2=0x7f2f080c52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:03.999 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.998+0000 7f2f0d9a1700 1 -- 172.21.15.27:0/1435697196 shutdown_connections 2023-10-06T01:42:03.999 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:03.998+0000 7f2f0d9a1700 1 -- 172.21.15.27:0/1435697196 wait complete. 2023-10-06T01:42:04.011 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T01:42:04.129 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.129+0000 7f1a405ed700 1 Processor -- start 2023-10-06T01:42:04.130 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.129+0000 7f1a405ed700 1 -- start start 2023-10-06T01:42:04.132 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.130+0000 7f1a405ed700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1a380cdc00 0x7f1a380cbe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:04.132 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.130+0000 7f1a405ed700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f1a380c6dc0 0x7f1a380ae5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:04.132 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.130+0000 7f1a405ed700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a380ae140 con 0x7f1a380d1030 2023-10-06T01:42:04.132 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.130+0000 7f1a405ed700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a380cc390 con 0x7f1a380cdc00 2023-10-06T01:42:04.132 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.130+0000 7f1a405ed700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a380ad530 con 0x7f1a380c6dc0 2023-10-06T01:42:04.132 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.130+0000 7f1a3e389700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1a380cdc00 0x7f1a380cbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:04.132 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.130+0000 7f1a3e389700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1a380cdc00 0x7f1a380cbe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54044/0 (socket says 172.21.15.27:54044) 2023-10-06T01:42:04.132 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.130+0000 7f1a3e389700 1 -- 172.21.15.27:0/465227588 learned_addr learned my addr 172.21.15.27:0/465227588 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:04.133 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.130+0000 7f1a3eb8a700 1 --2- 172.21.15.27:0/465227588 >> v2:172.21.15.124:3300/0 conn(0x7f1a380c6dc0 0x7f1a380ae5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:04.133 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.131+0000 7f1a3e389700 1 -- 172.21.15.27:0/465227588 >> v2:172.21.15.124:3300/0 conn(0x7f1a380c6dc0 msgr2=0x7f1a380ae5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:04.133 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.131+0000 7f1a3e389700 1 --2- 172.21.15.27:0/465227588 >> v2:172.21.15.124:3300/0 conn(0x7f1a380c6dc0 0x7f1a380ae5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:04.133 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.131+0000 7f1a3e389700 1 -- 172.21.15.27:0/465227588 >> v1:172.21.15.124:6789/0 conn(0x7f1a380d1030 legacy=0x7f1a380c66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:04.133 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.131+0000 7f1a3e389700 1 -- 172.21.15.27:0/465227588 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a380ad810 con 0x7f1a380cdc00 2023-10-06T01:42:04.133 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.131+0000 7f1a3e389700 1 --2- 172.21.15.27:0/465227588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a380cdc00 0x7f1a380cbe50 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f1a3808a900 tx=0x7f1a30009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce65550503cf86a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:04.133 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.131+0000 7f1a3d387700 1 -- 172.21.15.27:0/465227588 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a30005c10 con 0x7f1a380cdc00 2023-10-06T01:42:04.133 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.131+0000 7f1a3d387700 1 -- 172.21.15.27:0/465227588 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a30005d70 con 0x7f1a380cdc00 2023-10-06T01:42:04.133 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.131+0000 7f1a3d387700 1 -- 172.21.15.27:0/465227588 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a30006260 con 0x7f1a380cdc00 2023-10-06T01:42:04.133 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.131+0000 7f1a405ed700 1 -- 172.21.15.27:0/465227588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a380cdc00 msgr2=0x7f1a380cbe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:04.133 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.131+0000 7f1a405ed700 1 --2- 172.21.15.27:0/465227588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a380cdc00 0x7f1a380cbe50 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f1a3808a900 tx=0x7f1a30009220 comp rx=0 tx=0).stop 2023-10-06T01:42:04.133 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.132+0000 7f1a405ed700 1 -- 172.21.15.27:0/465227588 shutdown_connections 2023-10-06T01:42:04.133 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.132+0000 7f1a405ed700 1 --2- 172.21.15.27:0/465227588 >> v2:172.21.15.124:3300/0 conn(0x7f1a380c6dc0 0x7f1a380ae5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:04.134 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.132+0000 7f1a405ed700 1 --2- 172.21.15.27:0/465227588 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a380cdc00 0x7f1a380cbe50 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:04.134 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.132+0000 7f1a405ed700 1 -- 172.21.15.27:0/465227588 >> 172.21.15.27:0/465227588 conn(0x7f1a38099990 msgr2=0x7f1a380b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:04.134 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.132+0000 7f1a405ed700 1 -- 172.21.15.27:0/465227588 shutdown_connections 2023-10-06T01:42:04.134 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.132+0000 7f1a405ed700 1 -- 172.21.15.27:0/465227588 wait complete. 2023-10-06T01:42:04.136 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.133+0000 7f1a405ed700 1 Processor -- start 2023-10-06T01:42:04.136 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.133+0000 7f1a405ed700 1 -- start start 2023-10-06T01:42:04.136 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.133+0000 7f1a405ed700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1a380c6dc0 0x7f1a380b1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:04.136 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.133+0000 7f1a405ed700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a380cdc00 0x7f1a380aff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:04.136 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.133+0000 7f1a405ed700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1a380d1030 0x7f1a380b0440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:04.136 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.133+0000 7f1a405ed700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a380b0980 con 0x7f1a380cdc00 2023-10-06T01:42:04.137 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.133+0000 7f1a405ed700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a380b17d0 con 0x7f1a380c6dc0 2023-10-06T01:42:04.137 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.133+0000 7f1a405ed700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a38158a90 con 0x7f1a380d1030 2023-10-06T01:42:04.137 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.134+0000 7f1a3db88700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a380cdc00 0x7f1a380aff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:04.137 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.134+0000 7f1a3db88700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a380cdc00 0x7f1a380aff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54050/0 (socket says 172.21.15.27:54050) 2023-10-06T01:42:04.137 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.134+0000 7f1a3db88700 1 -- 172.21.15.27:0/2605111623 learned_addr learned my addr 172.21.15.27:0/2605111623 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:04.137 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.134+0000 7f1a3e389700 1 --2- 172.21.15.27:0/2605111623 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1a380c6dc0 0x7f1a380b1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:04.137 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.134+0000 7f1a3eb8a700 1 --2- 172.21.15.27:0/2605111623 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1a380d1030 0x7f1a380b0440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:04.137 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.134+0000 7f1a3db88700 1 -- 172.21.15.27:0/2605111623 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1a380d1030 msgr2=0x7f1a380b0440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:04.137 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.134+0000 7f1a3db88700 1 --2- 172.21.15.27:0/2605111623 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1a380d1030 0x7f1a380b0440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:04.137 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.134+0000 7f1a3db88700 1 -- 172.21.15.27:0/2605111623 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1a380c6dc0 msgr2=0x7f1a380b1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:04.137 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.134+0000 7f1a3db88700 1 --2- 172.21.15.27:0/2605111623 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1a380c6dc0 0x7f1a380b1c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:04.137 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.134+0000 7f1a3db88700 1 -- 172.21.15.27:0/2605111623 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a30009480 con 0x7f1a380cdc00 2023-10-06T01:42:04.138 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.134+0000 7f1a3db88700 1 --2- 172.21.15.27:0/2605111623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a380cdc00 0x7f1a380aff00 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c00c510 tx=0x7f1a2c00c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:04.138 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.134+0000 7f1a367fc700 1 -- 172.21.15.27:0/2605111623 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a2c011590 con 0x7f1a380cdc00 2023-10-06T01:42:04.138 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.134+0000 7f1a367fc700 1 -- 172.21.15.27:0/2605111623 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a2c011710 con 0x7f1a380cdc00 2023-10-06T01:42:04.138 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.134+0000 7f1a405ed700 1 -- 172.21.15.27:0/2605111623 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a380b0bc0 con 0x7f1a380cdc00 2023-10-06T01:42:04.138 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.134+0000 7f1a367fc700 1 -- 172.21.15.27:0/2605111623 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a2c011ad0 con 0x7f1a380cdc00 2023-10-06T01:42:04.138 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.134+0000 7f1a405ed700 1 -- 172.21.15.27:0/2605111623 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a380b1180 con 0x7f1a380cdc00 2023-10-06T01:42:04.138 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.135+0000 7f1a405ed700 1 -- 172.21.15.27:0/2605111623 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a380b1980 con 0x7f1a380cdc00 2023-10-06T01:42:04.138 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.136+0000 7f1a367fc700 1 -- 172.21.15.27:0/2605111623 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1a2c01b3b0 con 0x7f1a380cdc00 2023-10-06T01:42:04.138 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.136+0000 7f1a367fc700 1 --2- 172.21.15.27:0/2605111623 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1a24066f30 0x7f1a240693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:04.138 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.136+0000 7f1a367fc700 1 -- 172.21.15.27:0/2605111623 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f1a2c083a60 con 0x7f1a380cdc00 2023-10-06T01:42:04.138 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.136+0000 7f1a3e389700 1 --2- 172.21.15.27:0/2605111623 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1a24066f30 0x7f1a240693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:04.139 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.137+0000 7f1a3e389700 1 --2- 172.21.15.27:0/2605111623 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1a24066f30 0x7f1a240693e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1a3809a3b0 tx=0x7f1a30016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:04.140 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.140+0000 7f1a367fc700 1 -- 172.21.15.27:0/2605111623 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1a380b1980 con 0x7f1a380cdc00 2023-10-06T01:42:04.336 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.336+0000 7f1a405ed700 1 -- 172.21.15.27:0/2605111623 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a380b1980 con 0x7f1a380cdc00 2023-10-06T01:42:04.339 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.337+0000 7f1a367fc700 1 -- 172.21.15.27:0/2605111623 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+16879 (secure 0 0 0) 0x7f1a380b1980 con 0x7f1a380cdc00 2023-10-06T01:42:04.339 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:04.339 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":30,"fsid":"20a7bff0-74f8-4f2f-b6c6-899e804bc965","created":"2023-10-06T01:40:54.474215+0000","modified":"2023-10-06T01:42:03.036481+0000","last_up_change":"2023-10-06T01:41:05.242847+0000","last_in_change":"2023-10-06T01:40:56.890960+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-10-06T01:41:05.685386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-10-06T01:41:06.092995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-10-06T01:42:01.295053+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-10-06T01:42:02.005152+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b902c86d-800c-40c5-b152-b545e50d02c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":90644},{"type":"v1","addr":"172.21.15.27:6801","nonce":90644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":90644},{"type":"v1","addr":"172.21.15.27:6803","nonce":90644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":90644},{"type":"v1","addr":"172.21.15.27:6807","nonce":90644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":90644},{"type":"v1","addr":"172.21.15.27:6805","nonce":90644}]},"public_addr":"172.21.15.27:6801/90644","cluster_addr":"172.21.15.27:6803/90644","heartbeat_back_addr":"172.21.15.27:6807/90644","heartbeat_front_addr":"172.21.15.27:6805/90644","state":["exists","up"]},{"osd":1,"uuid":"5b4a70e5-b06e-48cd-bfc4-7962c5d05386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":90643},{"type":"v1","addr":"172.21.15.27:6809","nonce":90643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":90643},{"type":"v1","addr":"172.21.15.27:6811","nonce":90643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":90643},{"type":"v1","addr":"172.21.15.27:6815","nonce":90643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":90643},{"type":"v1","addr":"172.21.15.27:6813","nonce":90643}]},"public_addr":"172.21.15.27:6809/90643","cluster_addr":"172.21.15.27:6811/90643","heartbeat_back_addr":"172.21.15.27:6815/90643","heartbeat_front_addr":"172.21.15.27:6813/90643","state":["exists","up"]},{"osd":2,"uuid":"22d9de46-4073-4336-9122-5227e601eca0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":90645},{"type":"v1","addr":"172.21.15.27:6817","nonce":90645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":90645},{"type":"v1","addr":"172.21.15.27:6819","nonce":90645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":90645},{"type":"v1","addr":"172.21.15.27:6823","nonce":90645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":90645},{"type":"v1","addr":"172.21.15.27:6821","nonce":90645}]},"public_addr":"172.21.15.27:6817/90645","cluster_addr":"172.21.15.27:6819/90645","heartbeat_back_addr":"172.21.15.27:6823/90645","heartbeat_front_addr":"172.21.15.27:6821/90645","state":["exists","up"]},{"osd":3,"uuid":"438bb2d3-34e2-4b7c-9734-8360c1b67650","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":90668},{"type":"v1","addr":"172.21.15.27:6825","nonce":90668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":90668},{"type":"v1","addr":"172.21.15.27:6827","nonce":90668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":90668},{"type":"v1","addr":"172.21.15.27:6831","nonce":90668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":90668},{"type":"v1","addr":"172.21.15.27:6829","nonce":90668}]},"public_addr":"172.21.15.27:6825/90668","cluster_addr":"172.21.15.27:6827/90668","heartbeat_back_addr":"172.21.15.27:6831/90668","heartbeat_front_addr":"172.21.15.27:6829/90668","state":["exists","up"]},{"osd":4,"uuid":"e1956a8e-e0f8-4e7c-acbb-40037415a354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":90863},{"type":"v1","addr":"172.21.15.124:6801","nonce":90863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":90863},{"type":"v1","addr":"172.21.15.124:6803","nonce":90863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":90863},{"type":"v1","addr":"172.21.15.124:6807","nonce":90863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":90863},{"type":"v1","addr":"172.21.15.124:6805","nonce":90863}]},"public_addr":"172.21.15.124:6801/90863","cluster_addr":"172.21.15.124:6803/90863","heartbeat_back_addr":"172.21.15.124:6807/90863","heartbeat_front_addr":"172.21.15.124:6805/90863","state":["exists","up"]},{"osd":5,"uuid":"306c59d4-846d-4b73-b1c9-051bec9e9368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":90885},{"type":"v1","addr":"172.21.15.124:6817","nonce":90885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":90885},{"type":"v1","addr":"172.21.15.124:6819","nonce":90885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":90885},{"type":"v1","addr":"172.21.15.124:6823","nonce":90885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":90885},{"type":"v1","addr":"172.21.15.124:6821","nonce":90885}]},"public_addr":"172.21.15.124:6817/90885","cluster_addr":"172.21.15.124:6819/90885","heartbeat_back_addr":"172.21.15.124:6823/90885","heartbeat_front_addr":"172.21.15.124:6821/90885","state":["exists","up"]},{"osd":6,"uuid":"15463e53-ac8d-47d1-b4f6-c0d31f29e4a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":90864},{"type":"v1","addr":"172.21.15.124:6809","nonce":90864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":90864},{"type":"v1","addr":"172.21.15.124:6811","nonce":90864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":90864},{"type":"v1","addr":"172.21.15.124:6815","nonce":90864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":90864},{"type":"v1","addr":"172.21.15.124:6813","nonce":90864}]},"public_addr":"172.21.15.124:6809/90864","cluster_addr":"172.21.15.124:6811/90864","heartbeat_back_addr":"172.21.15.124:6815/90864","heartbeat_front_addr":"172.21.15.124:6813/90864","state":["exists","up"]},{"osd":7,"uuid":"ae463a90-a9d3-498f-86fa-47a35b3fe889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":90894},{"type":"v1","addr":"172.21.15.124:6825","nonce":90894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":90894},{"type":"v1","addr":"172.21.15.124:6827","nonce":90894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":90894},{"type":"v1","addr":"172.21.15.124:6831","nonce":90894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":90894},{"type":"v1","addr":"172.21.15.124:6829","nonce":90894}]},"public_addr":"172.21.15.124:6825/90894","cluster_addr":"172.21.15.124:6827/90894","heartbeat_back_addr":"172.21.15.124:6831/90894","heartbeat_front_addr":"172.21.15.124:6829/90894","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.827704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.839951+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.032230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.050605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.987908+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.031290+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.080707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.053692+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T01:42:04.340 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.339+0000 7f1a405ed700 1 -- 172.21.15.27:0/2605111623 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1a24066f30 msgr2=0x7f1a240693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:04.340 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.339+0000 7f1a405ed700 1 --2- 172.21.15.27:0/2605111623 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1a24066f30 0x7f1a240693e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1a3809a3b0 tx=0x7f1a30016000 comp rx=0 tx=0).stop 2023-10-06T01:42:04.340 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.339+0000 7f1a405ed700 1 -- 172.21.15.27:0/2605111623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a380cdc00 msgr2=0x7f1a380aff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:04.341 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.339+0000 7f1a405ed700 1 --2- 172.21.15.27:0/2605111623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a380cdc00 0x7f1a380aff00 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f1a2c00c510 tx=0x7f1a2c00c820 comp rx=0 tx=0).stop 2023-10-06T01:42:04.341 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.339+0000 7f1a405ed700 1 -- 172.21.15.27:0/2605111623 shutdown_connections 2023-10-06T01:42:04.341 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.339+0000 7f1a405ed700 1 --2- 172.21.15.27:0/2605111623 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1a24066f30 0x7f1a240693e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:04.341 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.339+0000 7f1a405ed700 1 --2- 172.21.15.27:0/2605111623 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1a380d1030 0x7f1a380b0440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:04.341 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.339+0000 7f1a405ed700 1 --2- 172.21.15.27:0/2605111623 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1a380c6dc0 0x7f1a380b1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:04.341 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.339+0000 7f1a405ed700 1 --2- 172.21.15.27:0/2605111623 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a380cdc00 0x7f1a380aff00 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:04.341 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.339+0000 7f1a405ed700 1 -- 172.21.15.27:0/2605111623 >> 172.21.15.27:0/2605111623 conn(0x7f1a38099990 msgr2=0x7f1a380b28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:04.341 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.339+0000 7f1a405ed700 1 -- 172.21.15.27:0/2605111623 shutdown_connections 2023-10-06T01:42:04.341 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.339+0000 7f1a405ed700 1 -- 172.21.15.27:0/2605111623 wait complete. 2023-10-06T01:42:04.352 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-06T01:42:04.474 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.473+0000 7f511ba1d700 1 Processor -- start 2023-10-06T01:42:04.474 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.473+0000 7f511ba1d700 1 -- start start 2023-10-06T01:42:04.477 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.474+0000 7f511ba1d700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f51140b03d0 0x7f51140a6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:04.477 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.474+0000 7f511ba1d700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51140ab680 con 0x7f51140b07a0 2023-10-06T01:42:04.477 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.474+0000 7f511ba1d700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f511415a630 con 0x7f5114156800 2023-10-06T01:42:04.477 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.474+0000 7f511ba1d700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f511415a910 con 0x7f51140b03d0 2023-10-06T01:42:04.477 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.474+0000 7f5118fb8700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f51140b07a0 0x7f51141560f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:54476/0 (socket says 172.21.15.27:54476) 2023-10-06T01:42:04.477 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.474+0000 7f5118fb8700 1 -- 172.21.15.27:0/3040787425 learned_addr learned my addr 172.21.15.27:0/3040787425 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:04.477 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.475+0000 7f5113fff700 1 -- 172.21.15.27:0/3040787425 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1152841891 0 0) 0x7f51140ab680 con 0x7f51140b07a0 2023-10-06T01:42:04.478 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.475+0000 7f5113fff700 1 -- 172.21.15.27:0/3040787425 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50fc003570 con 0x7f51140b07a0 2023-10-06T01:42:04.478 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.475+0000 7f51197b9700 1 --2- 172.21.15.27:0/3040787425 >> v2:172.21.15.124:3300/0 conn(0x7f51140b03d0 0x7f51140a6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:04.478 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.475+0000 7f5113fff700 1 -- 172.21.15.27:0/3040787425 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1826142524 0 0) 0x7f50fc003570 con 0x7f51140b07a0 2023-10-06T01:42:04.478 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.475+0000 7f5113fff700 1 -- 172.21.15.27:0/3040787425 >> v2:172.21.15.124:3300/0 conn(0x7f51140b03d0 msgr2=0x7f51140a6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:04.478 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.475+0000 7f5113fff700 1 --2- 172.21.15.27:0/3040787425 >> v2:172.21.15.124:3300/0 conn(0x7f51140b03d0 0x7f51140a6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:04.478 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.475+0000 7f5113fff700 1 -- 172.21.15.27:0/3040787425 >> v1:172.21.15.124:6789/0 conn(0x7f5114156800 legacy=0x7f51140ad0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:04.478 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.475+0000 7f5113fff700 1 -- 172.21.15.27:0/3040787425 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f511415abf0 con 0x7f51140b07a0 2023-10-06T01:42:04.478 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.475+0000 7f5113fff700 1 -- 172.21.15.27:0/3040787425 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2619078944 0 0) 0x7f5108001470 con 0x7f51140b07a0 2023-10-06T01:42:04.478 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.475+0000 7f5113fff700 1 -- 172.21.15.27:0/3040787425 >> v1:172.21.15.27:6789/0 conn(0x7f51140b07a0 legacy=0x7f51141560f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:04.478 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.475+0000 7f5113fff700 1 --2- 172.21.15.27:0/3040787425 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f50fc003a60 0x7f50fc003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:04.479 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.475+0000 7f5113fff700 1 --2- 172.21.15.27:0/3040787425 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50fc004810 0x7f50fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:04.479 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.475+0000 7f5113fff700 1 --2- 172.21.15.27:0/3040787425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50fc0051c0 0x7f50fc00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:04.479 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.475+0000 7f5113fff700 1 -- 172.21.15.27:0/3040787425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511415a910 con 0x7f50fc0051c0 2023-10-06T01:42:04.479 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.475+0000 7f5113fff700 1 -- 172.21.15.27:0/3040787425 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50fc00c160 con 0x7f50fc004810 2023-10-06T01:42:04.479 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.475+0000 7f5113fff700 1 -- 172.21.15.27:0/3040787425 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50fc00c320 con 0x7f50fc003a60 2023-10-06T01:42:04.479 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.475+0000 7f5119fba700 1 --2- 172.21.15.27:0/3040787425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50fc0051c0 0x7f50fc00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:04.479 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.475+0000 7f51197b9700 1 --2- 172.21.15.27:0/3040787425 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f50fc003a60 0x7f50fc003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:04.479 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.475+0000 7f5118fb8700 1 --2- 172.21.15.27:0/3040787425 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50fc004810 0x7f50fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:04.479 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.475+0000 7f5119fba700 1 -- 172.21.15.27:0/3040787425 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f50fc003a60 msgr2=0x7f50fc003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:04.479 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.475+0000 7f5119fba700 1 --2- 172.21.15.27:0/3040787425 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f50fc003a60 0x7f50fc003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:04.479 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.475+0000 7f5119fba700 1 -- 172.21.15.27:0/3040787425 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50fc004810 msgr2=0x7f50fc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.475+0000 7f5119fba700 1 --2- 172.21.15.27:0/3040787425 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50fc004810 0x7f50fc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.475+0000 7f5119fba700 1 -- 172.21.15.27:0/3040787425 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50fc003570 con 0x7f50fc0051c0 2023-10-06T01:42:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.476+0000 7f5119fba700 1 -- 172.21.15.27:0/3040787425 reap_dead start 2023-10-06T01:42:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.476+0000 7f5119fba700 1 --2- 172.21.15.27:0/3040787425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50fc0051c0 0x7f50fc00ba60 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5104001fc0 tx=0x7f5104001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49e6ec6907bb63c1 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.476+0000 7f5113fff700 1 -- 172.21.15.27:0/3040787425 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5104001950 con 0x7f50fc0051c0 2023-10-06T01:42:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.476+0000 7f5113fff700 1 -- 172.21.15.27:0/3040787425 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f510400c8b0 con 0x7f50fc0051c0 2023-10-06T01:42:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.476+0000 7f5113fff700 1 -- 172.21.15.27:0/3040787425 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f510400cc50 con 0x7f50fc0051c0 2023-10-06T01:42:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.476+0000 7f511ba1d700 1 -- 172.21.15.27:0/3040787425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50fc0051c0 msgr2=0x7f50fc00ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.476+0000 7f511ba1d700 1 --2- 172.21.15.27:0/3040787425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50fc0051c0 0x7f50fc00ba60 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5104001fc0 tx=0x7f5104001640 comp rx=0 tx=0).stop 2023-10-06T01:42:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.476+0000 7f511ba1d700 1 -- 172.21.15.27:0/3040787425 shutdown_connections 2023-10-06T01:42:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.476+0000 7f511ba1d700 1 --2- 172.21.15.27:0/3040787425 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f50fc0051c0 0x7f50fc00ba60 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:04.480 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.476+0000 7f511ba1d700 1 -- 172.21.15.27:0/3040787425 >> 172.21.15.27:0/3040787425 conn(0x7f51140680b0 msgr2=0x7f51140684b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:04.481 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.476+0000 7f511ba1d700 1 -- 172.21.15.27:0/3040787425 shutdown_connections 2023-10-06T01:42:04.481 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.476+0000 7f511ba1d700 1 -- 172.21.15.27:0/3040787425 wait complete. 2023-10-06T01:42:04.483 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.477+0000 7f511ba1d700 1 Processor -- start 2023-10-06T01:42:04.483 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.478+0000 7f511ba1d700 1 -- start start 2023-10-06T01:42:04.483 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.478+0000 7f511ba1d700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50fc0051c0 0x7f51141532c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:04.483 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.478+0000 7f511ba1d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5114155e50 0x7f5114147100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:04.484 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.478+0000 7f511ba1d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5114154a60 0x7f5114154ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:04.484 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.478+0000 7f511ba1d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51141545d0 con 0x7f5114155e50 2023-10-06T01:42:04.484 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.478+0000 7f511ba1d700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51141539c0 con 0x7f50fc0051c0 2023-10-06T01:42:04.484 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.478+0000 7f511ba1d700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5114153ca0 con 0x7f5114154a60 2023-10-06T01:42:04.484 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.479+0000 7f5118fb8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5114155e50 0x7f5114147100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:04.484 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.479+0000 7f5118fb8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5114155e50 0x7f5114147100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:54070/0 (socket says 172.21.15.27:54070) 2023-10-06T01:42:04.484 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.479+0000 7f5118fb8700 1 -- 172.21.15.27:0/2037721158 learned_addr learned my addr 172.21.15.27:0/2037721158 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:04.484 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.479+0000 7f51197b9700 1 --2- 172.21.15.27:0/2037721158 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50fc0051c0 0x7f51141532c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:04.484 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.479+0000 7f5119fba700 1 --2- 172.21.15.27:0/2037721158 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5114154a60 0x7f5114154ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:04.484 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.479+0000 7f5118fb8700 1 -- 172.21.15.27:0/2037721158 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5114154a60 msgr2=0x7f5114154ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:04.484 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.479+0000 7f5118fb8700 1 --2- 172.21.15.27:0/2037721158 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5114154a60 0x7f5114154ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:04.484 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.479+0000 7f5118fb8700 1 -- 172.21.15.27:0/2037721158 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50fc0051c0 msgr2=0x7f51141532c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:04.485 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.479+0000 7f5118fb8700 1 --2- 172.21.15.27:0/2037721158 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50fc0051c0 0x7f51141532c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:04.485 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.479+0000 7f5118fb8700 1 -- 172.21.15.27:0/2037721158 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5114153f80 con 0x7f5114155e50 2023-10-06T01:42:04.485 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.479+0000 7f5118fb8700 1 --2- 172.21.15.27:0/2037721158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5114155e50 0x7f5114147100 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5108007370 tx=0x7f5108001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:04.485 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.479+0000 7f51117fa700 1 -- 172.21.15.27:0/2037721158 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5108018030 con 0x7f5114155e50 2023-10-06T01:42:04.485 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.479+0000 7f51117fa700 1 -- 172.21.15.27:0/2037721158 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f510800f8d0 con 0x7f5114155e50 2023-10-06T01:42:04.485 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.479+0000 7f51117fa700 1 -- 172.21.15.27:0/2037721158 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f510800fc70 con 0x7f5114155e50 2023-10-06T01:42:04.485 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.479+0000 7f511ba1d700 1 -- 172.21.15.27:0/2037721158 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5114154260 con 0x7f5114155e50 2023-10-06T01:42:04.485 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.479+0000 7f511ba1d700 1 -- 172.21.15.27:0/2037721158 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5114155410 con 0x7f5114155e50 2023-10-06T01:42:04.485 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.480+0000 7f511ba1d700 1 -- 172.21.15.27:0/2037721158 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5114154740 con 0x7f5114155e50 2023-10-06T01:42:04.485 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.481+0000 7f51117fa700 1 -- 172.21.15.27:0/2037721158 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f510801b070 con 0x7f5114155e50 2023-10-06T01:42:04.485 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.481+0000 7f51117fa700 1 --2- 172.21.15.27:0/2037721158 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5100066f30 0x7f51000693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:04.485 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.481+0000 7f51117fa700 1 -- 172.21.15.27:0/2037721158 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f5108019070 con 0x7f5114155e50 2023-10-06T01:42:04.486 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.481+0000 7f51197b9700 1 --2- 172.21.15.27:0/2037721158 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5100066f30 0x7f51000693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:04.486 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.482+0000 7f51197b9700 1 --2- 172.21.15.27:0/2037721158 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5100066f30 0x7f51000693e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f510c006ed0 tx=0x7f510c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:04.486 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.485+0000 7f51117fa700 1 -- 172.21.15.27:0/2037721158 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f51080517d0 con 0x7f5114155e50 2023-10-06T01:42:04.725 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:04.725+0000 7f511ba1d700 1 -- 172.21.15.27:0/2037721158 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f51141548d0 con 0x7f5114155e50 2023-10-06T01:42:05.650 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.649+0000 7f51117fa700 1 -- 172.21.15.27:0/2037721158 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v14) v1 ==== 118+0+0 (secure 0 0 0) 0x7f5108054df0 con 0x7f5114155e50 2023-10-06T01:42:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.652+0000 7f511ba1d700 1 -- 172.21.15.27:0/2037721158 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5100066f30 msgr2=0x7f51000693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.652+0000 7f511ba1d700 1 --2- 172.21.15.27:0/2037721158 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5100066f30 0x7f51000693e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f510c006ed0 tx=0x7f510c008040 comp rx=0 tx=0).stop 2023-10-06T01:42:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.652+0000 7f511ba1d700 1 -- 172.21.15.27:0/2037721158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5114155e50 msgr2=0x7f5114147100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.652+0000 7f511ba1d700 1 --2- 172.21.15.27:0/2037721158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5114155e50 0x7f5114147100 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5108007370 tx=0x7f5108001e50 comp rx=0 tx=0).stop 2023-10-06T01:42:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.652+0000 7f511ba1d700 1 -- 172.21.15.27:0/2037721158 shutdown_connections 2023-10-06T01:42:05.653 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.652+0000 7f511ba1d700 1 --2- 172.21.15.27:0/2037721158 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5100066f30 0x7f51000693e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:05.654 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.652+0000 7f511ba1d700 1 --2- 172.21.15.27:0/2037721158 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f5114154a60 0x7f5114154ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:05.654 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.652+0000 7f511ba1d700 1 --2- 172.21.15.27:0/2037721158 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f50fc0051c0 0x7f51141532c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:05.654 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.652+0000 7f511ba1d700 1 --2- 172.21.15.27:0/2037721158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f5114155e50 0x7f5114147100 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:05.654 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.652+0000 7f511ba1d700 1 -- 172.21.15.27:0/2037721158 >> 172.21.15.27:0/2037721158 conn(0x7f51140680b0 msgr2=0x7f51140ae6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:05.654 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.652+0000 7f511ba1d700 1 -- 172.21.15.27:0/2037721158 shutdown_connections 2023-10-06T01:42:05.654 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.652+0000 7f511ba1d700 1 -- 172.21.15.27:0/2037721158 wait complete. 2023-10-06T01:42:05.668 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-06T01:42:05.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.795+0000 7f4c1b646700 1 Processor -- start 2023-10-06T01:42:05.797 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.796+0000 7f4c1b646700 1 -- start start 2023-10-06T01:42:05.800 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.796+0000 7f4c1b646700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f4c1415dab0 0x7f4c140cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:05.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.796+0000 7f4c1b646700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c140b1110 con 0x7f4c141598d0 2023-10-06T01:42:05.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.796+0000 7f4c1b646700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c140b13b0 con 0x7f4c14159500 2023-10-06T01:42:05.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.796+0000 7f4c1b646700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c140b1690 con 0x7f4c1415dab0 2023-10-06T01:42:05.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.797+0000 7f4c18be1700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4c141598d0 0x7f4c1415d3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52078/0 (socket says 172.21.15.27:52078) 2023-10-06T01:42:05.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.797+0000 7f4c18be1700 1 -- 172.21.15.27:0/1249425674 learned_addr learned my addr 172.21.15.27:0/1249425674 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:05.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.797+0000 7f4c19be3700 1 --2- 172.21.15.27:0/1249425674 >> v2:172.21.15.124:3300/0 conn(0x7f4c1415dab0 0x7f4c140cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:05.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.797+0000 7f4c13fff700 1 -- 172.21.15.27:0/1249425674 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 515078815 0 0) 0x7f4c140b1110 con 0x7f4c141598d0 2023-10-06T01:42:05.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.797+0000 7f4c13fff700 1 -- 172.21.15.27:0/1249425674 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bfc003570 con 0x7f4c141598d0 2023-10-06T01:42:05.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.797+0000 7f4c13fff700 1 -- 172.21.15.27:0/1249425674 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 250992487 0 0) 0x7f4bfc003570 con 0x7f4c141598d0 2023-10-06T01:42:05.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.797+0000 7f4c13fff700 1 -- 172.21.15.27:0/1249425674 >> v2:172.21.15.124:3300/0 conn(0x7f4c1415dab0 msgr2=0x7f4c140cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:05.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.797+0000 7f4c13fff700 1 --2- 172.21.15.27:0/1249425674 >> v2:172.21.15.124:3300/0 conn(0x7f4c1415dab0 0x7f4c140cf6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:05.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.797+0000 7f4c13fff700 1 -- 172.21.15.27:0/1249425674 >> v1:172.21.15.124:6789/0 conn(0x7f4c14159500 legacy=0x7f4c140b3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:05.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.797+0000 7f4c13fff700 1 -- 172.21.15.27:0/1249425674 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c140b1970 con 0x7f4c141598d0 2023-10-06T01:42:05.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.797+0000 7f4c13fff700 1 -- 172.21.15.27:0/1249425674 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2619078944 0 0) 0x7f4c08001470 con 0x7f4c141598d0 2023-10-06T01:42:05.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.797+0000 7f4c19be3700 1 --2- 172.21.15.27:0/1249425674 >> v2:172.21.15.124:3300/0 conn(0x7f4c1415dab0 0x7f4c140cf6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:42:05.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.797+0000 7f4c13fff700 1 -- 172.21.15.27:0/1249425674 >> v1:172.21.15.27:6789/0 conn(0x7f4c141598d0 legacy=0x7f4c1415d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:05.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.797+0000 7f4c13fff700 1 --2- 172.21.15.27:0/1249425674 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4bfc003a60 0x7f4bfc003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:05.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.797+0000 7f4c13fff700 1 --2- 172.21.15.27:0/1249425674 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4bfc004810 0x7f4bfc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:05.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.797+0000 7f4c13fff700 1 --2- 172.21.15.27:0/1249425674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bfc0051c0 0x7f4bfc00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:05.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.797+0000 7f4c13fff700 1 -- 172.21.15.27:0/1249425674 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c140b1690 con 0x7f4bfc0051c0 2023-10-06T01:42:05.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.797+0000 7f4c13fff700 1 -- 172.21.15.27:0/1249425674 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bfc00c0d0 con 0x7f4bfc004810 2023-10-06T01:42:05.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.797+0000 7f4c13fff700 1 -- 172.21.15.27:0/1249425674 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bfc00c290 con 0x7f4bfc003a60 2023-10-06T01:42:05.802 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.798+0000 7f4c19be3700 1 --2- 172.21.15.27:0/1249425674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bfc0051c0 0x7f4bfc00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:05.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.798+0000 7f4c18be1700 1 --2- 172.21.15.27:0/1249425674 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4bfc004810 0x7f4bfc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:05.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.798+0000 7f4c193e2700 1 --2- 172.21.15.27:0/1249425674 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4bfc003a60 0x7f4bfc003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:05.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.798+0000 7f4c19be3700 1 -- 172.21.15.27:0/1249425674 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4bfc003a60 msgr2=0x7f4bfc003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:05.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.798+0000 7f4c19be3700 1 --2- 172.21.15.27:0/1249425674 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4bfc003a60 0x7f4bfc003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:05.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.798+0000 7f4c19be3700 1 -- 172.21.15.27:0/1249425674 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4bfc004810 msgr2=0x7f4bfc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:05.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.798+0000 7f4c19be3700 1 --2- 172.21.15.27:0/1249425674 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4bfc004810 0x7f4bfc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:05.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.798+0000 7f4c19be3700 1 -- 172.21.15.27:0/1249425674 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bfc003570 con 0x7f4bfc0051c0 2023-10-06T01:42:05.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.798+0000 7f4c19be3700 1 -- 172.21.15.27:0/1249425674 reap_dead start 2023-10-06T01:42:05.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.798+0000 7f4c19be3700 1 --2- 172.21.15.27:0/1249425674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bfc0051c0 0x7f4bfc00ba60 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f4c1415deb0 tx=0x7f4c040091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7caa08310b8dcc server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:05.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.798+0000 7f4c13fff700 1 -- 172.21.15.27:0/1249425674 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c04009740 con 0x7f4bfc0051c0 2023-10-06T01:42:05.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.798+0000 7f4c13fff700 1 -- 172.21.15.27:0/1249425674 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c0400e920 con 0x7f4bfc0051c0 2023-10-06T01:42:05.803 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.798+0000 7f4c13fff700 1 -- 172.21.15.27:0/1249425674 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c0400ece0 con 0x7f4bfc0051c0 2023-10-06T01:42:05.804 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.799+0000 7f4c1b646700 1 -- 172.21.15.27:0/1249425674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bfc0051c0 msgr2=0x7f4bfc00ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:05.804 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.799+0000 7f4c1b646700 1 --2- 172.21.15.27:0/1249425674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bfc0051c0 0x7f4bfc00ba60 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f4c1415deb0 tx=0x7f4c040091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:05.804 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.799+0000 7f4c1b646700 1 -- 172.21.15.27:0/1249425674 shutdown_connections 2023-10-06T01:42:05.804 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.799+0000 7f4c1b646700 1 --2- 172.21.15.27:0/1249425674 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4bfc0051c0 0x7f4bfc00ba60 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:05.804 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.799+0000 7f4c1b646700 1 -- 172.21.15.27:0/1249425674 >> 172.21.15.27:0/1249425674 conn(0x7f4c14067630 msgr2=0x7f4c14069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:05.804 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.799+0000 7f4c1b646700 1 -- 172.21.15.27:0/1249425674 shutdown_connections 2023-10-06T01:42:05.804 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.799+0000 7f4c1b646700 1 -- 172.21.15.27:0/1249425674 wait complete. 2023-10-06T01:42:05.807 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.800+0000 7f4c1b646700 1 Processor -- start 2023-10-06T01:42:05.807 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.800+0000 7f4c1b646700 1 -- start start 2023-10-06T01:42:05.807 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.801+0000 7f4c1b646700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4bfc0051c0 0x7f4c1415d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:05.807 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.801+0000 7f4c1b646700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c14153d20 0x7f4c14150a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:05.807 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.801+0000 7f4c1b646700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4c14151f50 0x7f4c1413ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:05.807 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.801+0000 7f4c1b646700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c14150fc0 con 0x7f4c14153d20 2023-10-06T01:42:05.807 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.801+0000 7f4c1b646700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1415a050 con 0x7f4c14151f50 2023-10-06T01:42:05.807 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.801+0000 7f4c1b646700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c141511b0 con 0x7f4bfc0051c0 2023-10-06T01:42:05.807 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.801+0000 7f4c18be1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c14153d20 0x7f4c14150a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:05.807 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.801+0000 7f4c18be1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c14153d20 0x7f4c14150a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37394/0 (socket says 172.21.15.27:37394) 2023-10-06T01:42:05.807 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.801+0000 7f4c18be1700 1 -- 172.21.15.27:0/1347410089 learned_addr learned my addr 172.21.15.27:0/1347410089 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:05.808 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.801+0000 7f4c193e2700 1 --2- 172.21.15.27:0/1347410089 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4bfc0051c0 0x7f4c1415d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:05.808 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.801+0000 7f4c19be3700 1 --2- 172.21.15.27:0/1347410089 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4c14151f50 0x7f4c1413ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:05.808 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.801+0000 7f4c18be1700 1 -- 172.21.15.27:0/1347410089 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4bfc0051c0 msgr2=0x7f4c1415d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:05.808 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.801+0000 7f4c18be1700 1 --2- 172.21.15.27:0/1347410089 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4bfc0051c0 0x7f4c1415d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:05.808 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.801+0000 7f4c18be1700 1 -- 172.21.15.27:0/1347410089 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4c14151f50 msgr2=0x7f4c1413ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:05.808 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.801+0000 7f4c18be1700 1 --2- 172.21.15.27:0/1347410089 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4c14151f50 0x7f4c1413ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:05.808 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.801+0000 7f4c18be1700 1 -- 172.21.15.27:0/1347410089 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c14151490 con 0x7f4c14153d20 2023-10-06T01:42:05.808 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.801+0000 7f4c19be3700 1 --2- 172.21.15.27:0/1347410089 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4c14151f50 0x7f4c1413ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:42:05.808 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.801+0000 7f4c193e2700 1 --2- 172.21.15.27:0/1347410089 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4bfc0051c0 0x7f4c1415d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:42:05.808 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.801+0000 7f4c18be1700 1 --2- 172.21.15.27:0/1347410089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c14153d20 0x7f4c14150a80 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4c080073c0 tx=0x7f4c08001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:05.808 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.801+0000 7f4c117fa700 1 -- 172.21.15.27:0/1347410089 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c08018030 con 0x7f4c14153d20 2023-10-06T01:42:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.801+0000 7f4c117fa700 1 -- 172.21.15.27:0/1347410089 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c0800f8d0 con 0x7f4c14153d20 2023-10-06T01:42:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.802+0000 7f4c1b646700 1 -- 172.21.15.27:0/1347410089 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c1415a2a0 con 0x7f4c14153d20 2023-10-06T01:42:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.802+0000 7f4c117fa700 1 -- 172.21.15.27:0/1347410089 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c0800fc90 con 0x7f4c14153d20 2023-10-06T01:42:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.802+0000 7f4c1b646700 1 -- 172.21.15.27:0/1347410089 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c14152390 con 0x7f4c14153d20 2023-10-06T01:42:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.802+0000 7f4c1b646700 1 -- 172.21.15.27:0/1347410089 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c1415a7c0 con 0x7f4c14153d20 2023-10-06T01:42:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.803+0000 7f4c117fa700 1 -- 172.21.15.27:0/1347410089 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4c0801b070 con 0x7f4c14153d20 2023-10-06T01:42:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.803+0000 7f4c117fa700 1 --2- 172.21.15.27:0/1347410089 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4c00066f30 0x7f4c000693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.803+0000 7f4c117fa700 1 -- 172.21.15.27:0/1347410089 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f4c08019070 con 0x7f4c14153d20 2023-10-06T01:42:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.804+0000 7f4c193e2700 1 --2- 172.21.15.27:0/1347410089 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4c00066f30 0x7f4c000693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:05.809 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.804+0000 7f4c193e2700 1 --2- 172.21.15.27:0/1347410089 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4c00066f30 0x7f4c000693e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c002dd0 tx=0x7f4c0c001640 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:05.810 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:05.808+0000 7f4c117fa700 1 -- 172.21.15.27:0/1347410089 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4c1415a7c0 con 0x7f4c14153d20 2023-10-06T01:42:06.047 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.046+0000 7f4c1b646700 1 -- 172.21.15.27:0/1347410089 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f4c1415a7c0 con 0x7f4c14153d20 2023-10-06T01:42:06.657 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.656+0000 7f4c117fa700 1 -- 172.21.15.27:0/1347410089 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v15) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4c1415a7c0 con 0x7f4c14153d20 2023-10-06T01:42:06.660 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.658+0000 7f4c1b646700 1 -- 172.21.15.27:0/1347410089 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4c00066f30 msgr2=0x7f4c000693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:06.660 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.658+0000 7f4c1b646700 1 --2- 172.21.15.27:0/1347410089 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4c00066f30 0x7f4c000693e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4c0c002dd0 tx=0x7f4c0c001640 comp rx=0 tx=0).stop 2023-10-06T01:42:06.660 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.658+0000 7f4c1b646700 1 -- 172.21.15.27:0/1347410089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c14153d20 msgr2=0x7f4c14150a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:06.660 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.658+0000 7f4c1b646700 1 --2- 172.21.15.27:0/1347410089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c14153d20 0x7f4c14150a80 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4c080073c0 tx=0x7f4c08001e50 comp rx=0 tx=0).stop 2023-10-06T01:42:06.660 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.658+0000 7f4c1b646700 1 -- 172.21.15.27:0/1347410089 shutdown_connections 2023-10-06T01:42:06.660 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.658+0000 7f4c1b646700 1 --2- 172.21.15.27:0/1347410089 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4c00066f30 0x7f4c000693e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:06.660 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.658+0000 7f4c1b646700 1 --2- 172.21.15.27:0/1347410089 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f4bfc0051c0 0x7f4c1415d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:06.660 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.658+0000 7f4c1b646700 1 --2- 172.21.15.27:0/1347410089 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f4c14151f50 0x7f4c1413ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:06.661 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.658+0000 7f4c1b646700 1 --2- 172.21.15.27:0/1347410089 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4c14153d20 0x7f4c14150a80 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:06.661 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.658+0000 7f4c1b646700 1 -- 172.21.15.27:0/1347410089 >> 172.21.15.27:0/1347410089 conn(0x7f4c14067630 msgr2=0x7f4c14156530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:06.661 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.658+0000 7f4c1b646700 1 -- 172.21.15.27:0/1347410089 shutdown_connections 2023-10-06T01:42:06.661 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.658+0000 7f4c1b646700 1 -- 172.21.15.27:0/1347410089 wait complete. 2023-10-06T01:42:06.672 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T01:42:06.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.789+0000 7f7a16728700 1 Processor -- start 2023-10-06T01:42:06.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.790+0000 7f7a16728700 1 -- start start 2023-10-06T01:42:06.793 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.790+0000 7f7a16728700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7a1006f5f0 0x7f7a101569d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:06.793 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.790+0000 7f7a16728700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7a100cc570 0x7f7a100adec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:06.793 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.790+0000 7f7a16728700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a100ae400 con 0x7f7a100cc1a0 2023-10-06T01:42:06.793 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.790+0000 7f7a16728700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a1014d990 con 0x7f7a100cc570 2023-10-06T01:42:06.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.790+0000 7f7a16728700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a10157020 con 0x7f7a1006f5f0 2023-10-06T01:42:06.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.790+0000 7f7a14cc5700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7a100cc570 0x7f7a100adec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:06.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.791+0000 7f7a14cc5700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7a100cc570 0x7f7a100adec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37406/0 (socket says 172.21.15.27:37406) 2023-10-06T01:42:06.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.791+0000 7f7a0ffff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7a100cc1a0 0x7f7a100ad7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52082/0 (socket says 172.21.15.27:52082) 2023-10-06T01:42:06.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.791+0000 7f7a14cc5700 1 -- 172.21.15.27:0/1757309714 learned_addr learned my addr 172.21.15.27:0/1757309714 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:06.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.791+0000 7f7a0f7fe700 1 --2- 172.21.15.27:0/1757309714 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7a1006f5f0 0x7f7a101569d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:06.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.791+0000 7f7a14cc5700 1 -- 172.21.15.27:0/1757309714 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7a1006f5f0 msgr2=0x7f7a101569d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:06.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.791+0000 7f7a14cc5700 1 --2- 172.21.15.27:0/1757309714 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7a1006f5f0 0x7f7a101569d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:06.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.791+0000 7f7a14cc5700 1 -- 172.21.15.27:0/1757309714 >> v1:172.21.15.27:6789/0 conn(0x7f7a100cc1a0 legacy=0x7f7a100ad7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:06.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.791+0000 7f7a0effd700 1 -- 172.21.15.27:0/1757309714 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 964754906 0 0) 0x7f7a100ae400 con 0x7f7a100cc1a0 2023-10-06T01:42:06.794 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.791+0000 7f7a14cc5700 1 -- 172.21.15.27:0/1757309714 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a10157300 con 0x7f7a100cc570 2023-10-06T01:42:06.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.791+0000 7f7a14cc5700 1 --2- 172.21.15.27:0/1757309714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a100cc570 0x7f7a100adec0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7a08009f50 tx=0x7f7a08009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40cddab4126da3c server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:06.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.791+0000 7f7a0effd700 1 -- 172.21.15.27:0/1757309714 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0800e070 con 0x7f7a100cc570 2023-10-06T01:42:06.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.791+0000 7f7a0effd700 1 -- 172.21.15.27:0/1757309714 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a0800ba70 con 0x7f7a100cc570 2023-10-06T01:42:06.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.791+0000 7f7a0effd700 1 -- 172.21.15.27:0/1757309714 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a080092e0 con 0x7f7a100cc570 2023-10-06T01:42:06.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.792+0000 7f7a16728700 1 -- 172.21.15.27:0/1757309714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a100cc570 msgr2=0x7f7a100adec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:06.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.792+0000 7f7a16728700 1 --2- 172.21.15.27:0/1757309714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a100cc570 0x7f7a100adec0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7a08009f50 tx=0x7f7a08009840 comp rx=0 tx=0).stop 2023-10-06T01:42:06.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.792+0000 7f7a16728700 1 -- 172.21.15.27:0/1757309714 shutdown_connections 2023-10-06T01:42:06.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.792+0000 7f7a16728700 1 --2- 172.21.15.27:0/1757309714 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a100cc570 0x7f7a100adec0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:06.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.792+0000 7f7a16728700 1 --2- 172.21.15.27:0/1757309714 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7a1006f5f0 0x7f7a101569d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:06.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.792+0000 7f7a16728700 1 -- 172.21.15.27:0/1757309714 >> 172.21.15.27:0/1757309714 conn(0x7f7a1006e6a0 msgr2=0x7f7a1006eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:06.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.792+0000 7f7a16728700 1 -- 172.21.15.27:0/1757309714 shutdown_connections 2023-10-06T01:42:06.795 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.792+0000 7f7a16728700 1 -- 172.21.15.27:0/1757309714 wait complete. 2023-10-06T01:42:06.798 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.793+0000 7f7a16728700 1 Processor -- start 2023-10-06T01:42:06.798 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.793+0000 7f7a16728700 1 -- start start 2023-10-06T01:42:06.799 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.794+0000 7f7a16728700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a1006f5f0 0x7f7a10169e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:06.799 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.794+0000 7f7a16728700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7a100cc1a0 0x7f7a100c6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:06.799 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.794+0000 7f7a16728700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7a100cc570 0x7f7a1016d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:06.799 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.794+0000 7f7a16728700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a100b0110 con 0x7f7a1006f5f0 2023-10-06T01:42:06.799 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.794+0000 7f7a16728700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a100b03b0 con 0x7f7a100cc1a0 2023-10-06T01:42:06.799 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.794+0000 7f7a16728700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1016d930 con 0x7f7a100cc570 2023-10-06T01:42:06.799 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.794+0000 7f7a0ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a1006f5f0 0x7f7a10169e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:06.799 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.794+0000 7f7a0ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a1006f5f0 0x7f7a10169e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37414/0 (socket says 172.21.15.27:37414) 2023-10-06T01:42:06.799 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.794+0000 7f7a0ffff700 1 -- 172.21.15.27:0/1762144561 learned_addr learned my addr 172.21.15.27:0/1762144561 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:06.799 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.794+0000 7f7a14cc5700 1 --2- 172.21.15.27:0/1762144561 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7a100cc570 0x7f7a1016d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:06.799 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.794+0000 7f7a0ffff700 1 -- 172.21.15.27:0/1762144561 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7a100cc570 msgr2=0x7f7a1016d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:06.799 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.794+0000 7f7a0ffff700 1 --2- 172.21.15.27:0/1762144561 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7a100cc570 0x7f7a1016d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:06.800 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.794+0000 7f7a0f7fe700 1 --2- 172.21.15.27:0/1762144561 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7a100cc1a0 0x7f7a100c6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:06.800 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.794+0000 7f7a0ffff700 1 -- 172.21.15.27:0/1762144561 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7a100cc1a0 msgr2=0x7f7a100c6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:06.800 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.794+0000 7f7a0ffff700 1 --2- 172.21.15.27:0/1762144561 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7a100cc1a0 0x7f7a100c6880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:06.800 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.794+0000 7f7a0ffff700 1 -- 172.21.15.27:0/1762144561 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a08009ae0 con 0x7f7a1006f5f0 2023-10-06T01:42:06.800 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.794+0000 7f7a14cc5700 1 --2- 172.21.15.27:0/1762144561 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7a100cc570 0x7f7a1016d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:42:06.800 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.794+0000 7f7a0ffff700 1 --2- 172.21.15.27:0/1762144561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a1006f5f0 0x7f7a10169e70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f7a040011e0 tx=0x7f7a04007e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:06.800 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.794+0000 7f79f7fff700 1 -- 172.21.15.27:0/1762144561 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a04007300 con 0x7f7a1006f5f0 2023-10-06T01:42:06.800 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.794+0000 7f79f7fff700 1 -- 172.21.15.27:0/1762144561 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a04007460 con 0x7f7a1006f5f0 2023-10-06T01:42:06.800 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.794+0000 7f7a16728700 1 -- 172.21.15.27:0/1762144561 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a1016db60 con 0x7f7a1006f5f0 2023-10-06T01:42:06.800 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.794+0000 7f79f7fff700 1 -- 172.21.15.27:0/1762144561 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a04004830 con 0x7f7a1006f5f0 2023-10-06T01:42:06.800 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.794+0000 7f7a16728700 1 -- 172.21.15.27:0/1762144561 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a1016e120 con 0x7f7a1006f5f0 2023-10-06T01:42:06.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.795+0000 7f7a16728700 1 -- 172.21.15.27:0/1762144561 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a10064050 con 0x7f7a1006f5f0 2023-10-06T01:42:06.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.796+0000 7f79f7fff700 1 -- 172.21.15.27:0/1762144561 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7a04004be0 con 0x7f7a1006f5f0 2023-10-06T01:42:06.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.796+0000 7f79f7fff700 1 --2- 172.21.15.27:0/1762144561 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f79fc066f30 0x7f79fc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:06.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.796+0000 7f79f7fff700 1 -- 172.21.15.27:0/1762144561 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f7a04083c70 con 0x7f7a1006f5f0 2023-10-06T01:42:06.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.796+0000 7f7a0f7fe700 1 --2- 172.21.15.27:0/1762144561 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f79fc066f30 0x7f79fc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:06.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.797+0000 7f7a0f7fe700 1 --2- 172.21.15.27:0/1762144561 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f79fc066f30 0x7f79fc0693e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7a100c70c0 tx=0x7f7a00008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:06.801 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:06.800+0000 7f79f7fff700 1 -- 172.21.15.27:0/1762144561 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7a04050d60 con 0x7f7a1006f5f0 2023-10-06T01:42:07.039 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.038+0000 7f7a16728700 1 -- 172.21.15.27:0/1762144561 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a100b0690 con 0x7f7a1006f5f0 2023-10-06T01:42:07.039 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.038+0000 7f79f7fff700 1 -- 172.21.15.27:0/1762144561 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4597 (secure 0 0 0) 0x7f7a04018020 con 0x7f7a1006f5f0 2023-10-06T01:42:07.040 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:07.040 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4870,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6836/1043044736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":1043044736},{"type":"v1","addr":"172.21.15.124:6836","nonce":1043044736}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4962,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/1207987117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1207987117},{"type":"v1","addr":"172.21.15.27:6834","nonce":1207987117}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4965,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/943261840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":943261840},{"type":"v1","addr":"172.21.15.27:6835","nonce":943261840}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T01:42:03.036784+0000","modified":"2023-10-06T01:42:06.652163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4974},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4974":{"gid":4974,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.124:6837/325247112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6835","nonce":325247112},{"type":"v1","addr":"172.21.15.124:6837","nonce":325247112}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-10-06T01:42:07.042 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.040+0000 7f7a16728700 1 -- 172.21.15.27:0/1762144561 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f79fc066f30 msgr2=0x7f79fc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:07.043 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.040+0000 7f7a16728700 1 --2- 172.21.15.27:0/1762144561 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f79fc066f30 0x7f79fc0693e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7a100c70c0 tx=0x7f7a00008040 comp rx=0 tx=0).stop 2023-10-06T01:42:07.043 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.041+0000 7f7a16728700 1 -- 172.21.15.27:0/1762144561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a1006f5f0 msgr2=0x7f7a10169e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:07.043 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.041+0000 7f7a16728700 1 --2- 172.21.15.27:0/1762144561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a1006f5f0 0x7f7a10169e70 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f7a040011e0 tx=0x7f7a04007e10 comp rx=0 tx=0).stop 2023-10-06T01:42:07.043 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.041+0000 7f7a16728700 1 -- 172.21.15.27:0/1762144561 shutdown_connections 2023-10-06T01:42:07.043 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.041+0000 7f7a16728700 1 --2- 172.21.15.27:0/1762144561 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f79fc066f30 0x7f79fc0693e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:07.043 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.041+0000 7f7a16728700 1 --2- 172.21.15.27:0/1762144561 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7a100cc570 0x7f7a1016d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:07.043 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.041+0000 7f7a16728700 1 --2- 172.21.15.27:0/1762144561 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a1006f5f0 0x7f7a10169e70 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:07.045 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.041+0000 7f7a16728700 1 --2- 172.21.15.27:0/1762144561 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7a100cc1a0 0x7f7a100c6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:07.045 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.041+0000 7f7a16728700 1 -- 172.21.15.27:0/1762144561 >> 172.21.15.27:0/1762144561 conn(0x7f7a1006e6a0 msgr2=0x7f7a100ad320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:07.045 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.041+0000 7f7a16728700 1 -- 172.21.15.27:0/1762144561 shutdown_connections 2023-10-06T01:42:07.045 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.041+0000 7f7a16728700 1 -- 172.21.15.27:0/1762144561 wait complete. 2023-10-06T01:42:07.045 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 15 2023-10-06T01:42:07.054 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T01:42:07.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.173+0000 7f1a35b9d700 1 Processor -- start 2023-10-06T01:42:07.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.173+0000 7f1a35b9d700 1 -- start start 2023-10-06T01:42:07.176 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.173+0000 7f1a35b9d700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f1a3015d480 0x7f1a30069860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:07.176 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.173+0000 7f1a35b9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1a300b0a20 0x7f1a300b5c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:07.176 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.173+0000 7f1a35b9d700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a300687c0 con 0x7f1a30069da0 2023-10-06T01:42:07.176 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.173+0000 7f1a35b9d700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a30068aa0 con 0x7f1a300b0a20 2023-10-06T01:42:07.176 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.173+0000 7f1a35b9d700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a30068d80 con 0x7f1a3015d480 2023-10-06T01:42:07.176 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.174+0000 7f1a3539c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1a300b0a20 0x7f1a300b5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:07.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.174+0000 7f1a3539c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f1a300b0a20 0x7f1a300b5c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37420/0 (socket says 172.21.15.27:37420) 2023-10-06T01:42:07.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.174+0000 7f1a3539c700 1 -- 172.21.15.27:0/1021321466 learned_addr learned my addr 172.21.15.27:0/1021321466 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:07.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.174+0000 7f1a34b9b700 1 --2- 172.21.15.27:0/1021321466 >> v2:172.21.15.124:3300/0 conn(0x7f1a3015d480 0x7f1a30069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:07.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.174+0000 7f1a3539c700 1 -- 172.21.15.27:0/1021321466 >> v2:172.21.15.124:3300/0 conn(0x7f1a3015d480 msgr2=0x7f1a30069860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:07.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.174+0000 7f1a3539c700 1 --2- 172.21.15.27:0/1021321466 >> v2:172.21.15.124:3300/0 conn(0x7f1a3015d480 0x7f1a30069860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:07.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.174+0000 7f1a3539c700 1 -- 172.21.15.27:0/1021321466 >> v1:172.21.15.124:6789/0 conn(0x7f1a30069da0 legacy=0x7f1a300b0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:07.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.174+0000 7f1a3539c700 1 -- 172.21.15.27:0/1021321466 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a30069060 con 0x7f1a300b0a20 2023-10-06T01:42:07.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.174+0000 7f1a3539c700 1 --2- 172.21.15.27:0/1021321466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a300b0a20 0x7f1a300b5c30 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1a30071370 tx=0x7f1a18009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e1ad3eb0e333585 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:07.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.174+0000 7f1a2f7fe700 1 -- 172.21.15.27:0/1021321466 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a1800b860 con 0x7f1a300b0a20 2023-10-06T01:42:07.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.174+0000 7f1a2f7fe700 1 -- 172.21.15.27:0/1021321466 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a1800b9c0 con 0x7f1a300b0a20 2023-10-06T01:42:07.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.174+0000 7f1a2f7fe700 1 -- 172.21.15.27:0/1021321466 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a1800bd60 con 0x7f1a300b0a20 2023-10-06T01:42:07.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.175+0000 7f1a35b9d700 1 -- 172.21.15.27:0/1021321466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a300b0a20 msgr2=0x7f1a300b5c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:07.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.175+0000 7f1a35b9d700 1 --2- 172.21.15.27:0/1021321466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a300b0a20 0x7f1a300b5c30 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f1a30071370 tx=0x7f1a18009250 comp rx=0 tx=0).stop 2023-10-06T01:42:07.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.175+0000 7f1a35b9d700 1 -- 172.21.15.27:0/1021321466 shutdown_connections 2023-10-06T01:42:07.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.175+0000 7f1a35b9d700 1 --2- 172.21.15.27:0/1021321466 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a300b0a20 0x7f1a300b5c30 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:07.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.175+0000 7f1a35b9d700 1 --2- 172.21.15.27:0/1021321466 >> v2:172.21.15.124:3300/0 conn(0x7f1a3015d480 0x7f1a30069860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:07.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.175+0000 7f1a35b9d700 1 -- 172.21.15.27:0/1021321466 >> 172.21.15.27:0/1021321466 conn(0x7f1a300ccf80 msgr2=0x7f1a300cd380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:07.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.175+0000 7f1a35b9d700 1 -- 172.21.15.27:0/1021321466 shutdown_connections 2023-10-06T01:42:07.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.175+0000 7f1a35b9d700 1 -- 172.21.15.27:0/1021321466 wait complete. 2023-10-06T01:42:07.180 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.176+0000 7f1a35b9d700 1 Processor -- start 2023-10-06T01:42:07.180 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.176+0000 7f1a35b9d700 1 -- start start 2023-10-06T01:42:07.180 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.177+0000 7f1a35b9d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1a30069da0 0x7f1a30140660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:07.180 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.177+0000 7f1a35b9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a300b0a20 0x7f1a300c7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:07.180 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.177+0000 7f1a35b9d700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1a3015d480 0x7f1a300cadd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:07.181 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.177+0000 7f1a35b9d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a300c63b0 con 0x7f1a300b0a20 2023-10-06T01:42:07.181 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.177+0000 7f1a35b9d700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a300c6690 con 0x7f1a3015d480 2023-10-06T01:42:07.181 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.177+0000 7f1a35b9d700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a300c6970 con 0x7f1a30069da0 2023-10-06T01:42:07.181 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.177+0000 7f1a2ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a300b0a20 0x7f1a300c7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:07.181 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.177+0000 7f1a2ffff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a300b0a20 0x7f1a300c7450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37422/0 (socket says 172.21.15.27:37422) 2023-10-06T01:42:07.181 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.177+0000 7f1a2ffff700 1 -- 172.21.15.27:0/246569149 learned_addr learned my addr 172.21.15.27:0/246569149 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:07.181 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.177+0000 7f1a34b9b700 1 --2- 172.21.15.27:0/246569149 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1a30069da0 0x7f1a30140660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:07.181 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.177+0000 7f1a3539c700 1 --2- 172.21.15.27:0/246569149 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1a3015d480 0x7f1a300cadd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:07.181 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.177+0000 7f1a2ffff700 1 -- 172.21.15.27:0/246569149 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1a30069da0 msgr2=0x7f1a30140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:07.181 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.177+0000 7f1a2ffff700 1 --2- 172.21.15.27:0/246569149 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1a30069da0 0x7f1a30140660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:07.181 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.177+0000 7f1a2ffff700 1 -- 172.21.15.27:0/246569149 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1a3015d480 msgr2=0x7f1a300cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:07.181 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.177+0000 7f1a2ffff700 1 --2- 172.21.15.27:0/246569149 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1a3015d480 0x7f1a300cadd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:07.182 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.177+0000 7f1a2ffff700 1 -- 172.21.15.27:0/246569149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a180094f0 con 0x7f1a300b0a20 2023-10-06T01:42:07.182 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.177+0000 7f1a2ffff700 1 --2- 172.21.15.27:0/246569149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a300b0a20 0x7f1a300c7450 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1a2000c590 tx=0x7f1a2000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:07.182 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.178+0000 7f1a2cff9700 1 -- 172.21.15.27:0/246569149 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a20011590 con 0x7f1a300b0a20 2023-10-06T01:42:07.182 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.178+0000 7f1a2cff9700 1 -- 172.21.15.27:0/246569149 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a200116f0 con 0x7f1a300b0a20 2023-10-06T01:42:07.182 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.178+0000 7f1a35b9d700 1 -- 172.21.15.27:0/246569149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a300c6c50 con 0x7f1a300b0a20 2023-10-06T01:42:07.182 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.178+0000 7f1a2cff9700 1 -- 172.21.15.27:0/246569149 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a20011ab0 con 0x7f1a300b0a20 2023-10-06T01:42:07.182 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.178+0000 7f1a35b9d700 1 -- 172.21.15.27:0/246569149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a30140ba0 con 0x7f1a300b0a20 2023-10-06T01:42:07.182 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.179+0000 7f1a35b9d700 1 -- 172.21.15.27:0/246569149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a3004c3c0 con 0x7f1a300b0a20 2023-10-06T01:42:07.182 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.179+0000 7f1a2cff9700 1 -- 172.21.15.27:0/246569149 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1a2001b3b0 con 0x7f1a300b0a20 2023-10-06T01:42:07.183 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.180+0000 7f1a2cff9700 1 --2- 172.21.15.27:0/246569149 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1a10066f30 0x7f1a100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:07.183 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.180+0000 7f1a2cff9700 1 -- 172.21.15.27:0/246569149 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f1a20083c40 con 0x7f1a300b0a20 2023-10-06T01:42:07.183 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.180+0000 7f1a34b9b700 1 --2- 172.21.15.27:0/246569149 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1a10066f30 0x7f1a100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:07.183 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.180+0000 7f1a34b9b700 1 --2- 172.21.15.27:0/246569149 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1a10066f30 0x7f1a100693e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1a280074f0 tx=0x7f1a28009700 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:07.185 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.184+0000 7f1a2cff9700 1 -- 172.21.15.27:0/246569149 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1a20050d50 con 0x7f1a300b0a20 2023-10-06T01:42:07.379 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.378+0000 7f1a35b9d700 1 -- 172.21.15.27:0/246569149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a30069340 con 0x7f1a300b0a20 2023-10-06T01:42:07.380 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.379+0000 7f1a2cff9700 1 -- 172.21.15.27:0/246569149 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) v1 ==== 74+0+16866 (secure 0 0 0) 0x7f1a20017020 con 0x7f1a300b0a20 2023-10-06T01:42:07.380 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:07.381 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":31,"fsid":"20a7bff0-74f8-4f2f-b6c6-899e804bc965","created":"2023-10-06T01:40:54.474215+0000","modified":"2023-10-06T01:42:04.639589+0000","last_up_change":"2023-10-06T01:41:05.242847+0000","last_in_change":"2023-10-06T01:40:56.890960+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-10-06T01:41:05.685386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-10-06T01:41:06.092995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-10-06T01:42:01.295053+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-10-06T01:42:02.005152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b902c86d-800c-40c5-b152-b545e50d02c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":90644},{"type":"v1","addr":"172.21.15.27:6801","nonce":90644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":90644},{"type":"v1","addr":"172.21.15.27:6803","nonce":90644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":90644},{"type":"v1","addr":"172.21.15.27:6807","nonce":90644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":90644},{"type":"v1","addr":"172.21.15.27:6805","nonce":90644}]},"public_addr":"172.21.15.27:6801/90644","cluster_addr":"172.21.15.27:6803/90644","heartbeat_back_addr":"172.21.15.27:6807/90644","heartbeat_front_addr":"172.21.15.27:6805/90644","state":["exists","up"]},{"osd":1,"uuid":"5b4a70e5-b06e-48cd-bfc4-7962c5d05386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":90643},{"type":"v1","addr":"172.21.15.27:6809","nonce":90643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":90643},{"type":"v1","addr":"172.21.15.27:6811","nonce":90643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":90643},{"type":"v1","addr":"172.21.15.27:6815","nonce":90643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":90643},{"type":"v1","addr":"172.21.15.27:6813","nonce":90643}]},"public_addr":"172.21.15.27:6809/90643","cluster_addr":"172.21.15.27:6811/90643","heartbeat_back_addr":"172.21.15.27:6815/90643","heartbeat_front_addr":"172.21.15.27:6813/90643","state":["exists","up"]},{"osd":2,"uuid":"22d9de46-4073-4336-9122-5227e601eca0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":90645},{"type":"v1","addr":"172.21.15.27:6817","nonce":90645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":90645},{"type":"v1","addr":"172.21.15.27:6819","nonce":90645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":90645},{"type":"v1","addr":"172.21.15.27:6823","nonce":90645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":90645},{"type":"v1","addr":"172.21.15.27:6821","nonce":90645}]},"public_addr":"172.21.15.27:6817/90645","cluster_addr":"172.21.15.27:6819/90645","heartbeat_back_addr":"172.21.15.27:6823/90645","heartbeat_front_addr":"172.21.15.27:6821/90645","state":["exists","up"]},{"osd":3,"uuid":"438bb2d3-34e2-4b7c-9734-8360c1b67650","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":90668},{"type":"v1","addr":"172.21.15.27:6825","nonce":90668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":90668},{"type":"v1","addr":"172.21.15.27:6827","nonce":90668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":90668},{"type":"v1","addr":"172.21.15.27:6831","nonce":90668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":90668},{"type":"v1","addr":"172.21.15.27:6829","nonce":90668}]},"public_addr":"172.21.15.27:6825/90668","cluster_addr":"172.21.15.27:6827/90668","heartbeat_back_addr":"172.21.15.27:6831/90668","heartbeat_front_addr":"172.21.15.27:6829/90668","state":["exists","up"]},{"osd":4,"uuid":"e1956a8e-e0f8-4e7c-acbb-40037415a354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":90863},{"type":"v1","addr":"172.21.15.124:6801","nonce":90863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":90863},{"type":"v1","addr":"172.21.15.124:6803","nonce":90863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":90863},{"type":"v1","addr":"172.21.15.124:6807","nonce":90863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":90863},{"type":"v1","addr":"172.21.15.124:6805","nonce":90863}]},"public_addr":"172.21.15.124:6801/90863","cluster_addr":"172.21.15.124:6803/90863","heartbeat_back_addr":"172.21.15.124:6807/90863","heartbeat_front_addr":"172.21.15.124:6805/90863","state":["exists","up"]},{"osd":5,"uuid":"306c59d4-846d-4b73-b1c9-051bec9e9368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":90885},{"type":"v1","addr":"172.21.15.124:6817","nonce":90885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":90885},{"type":"v1","addr":"172.21.15.124:6819","nonce":90885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":90885},{"type":"v1","addr":"172.21.15.124:6823","nonce":90885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":90885},{"type":"v1","addr":"172.21.15.124:6821","nonce":90885}]},"public_addr":"172.21.15.124:6817/90885","cluster_addr":"172.21.15.124:6819/90885","heartbeat_back_addr":"172.21.15.124:6823/90885","heartbeat_front_addr":"172.21.15.124:6821/90885","state":["exists","up"]},{"osd":6,"uuid":"15463e53-ac8d-47d1-b4f6-c0d31f29e4a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":90864},{"type":"v1","addr":"172.21.15.124:6809","nonce":90864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":90864},{"type":"v1","addr":"172.21.15.124:6811","nonce":90864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":90864},{"type":"v1","addr":"172.21.15.124:6815","nonce":90864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":90864},{"type":"v1","addr":"172.21.15.124:6813","nonce":90864}]},"public_addr":"172.21.15.124:6809/90864","cluster_addr":"172.21.15.124:6811/90864","heartbeat_back_addr":"172.21.15.124:6815/90864","heartbeat_front_addr":"172.21.15.124:6813/90864","state":["exists","up"]},{"osd":7,"uuid":"ae463a90-a9d3-498f-86fa-47a35b3fe889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":90894},{"type":"v1","addr":"172.21.15.124:6825","nonce":90894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":90894},{"type":"v1","addr":"172.21.15.124:6827","nonce":90894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":90894},{"type":"v1","addr":"172.21.15.124:6831","nonce":90894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":90894},{"type":"v1","addr":"172.21.15.124:6829","nonce":90894}]},"public_addr":"172.21.15.124:6825/90894","cluster_addr":"172.21.15.124:6827/90894","heartbeat_back_addr":"172.21.15.124:6831/90894","heartbeat_front_addr":"172.21.15.124:6829/90894","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.827704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.839951+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.032230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.050605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.987908+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.031290+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.080707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.053692+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T01:42:07.383 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.381+0000 7f1a35b9d700 1 -- 172.21.15.27:0/246569149 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1a10066f30 msgr2=0x7f1a100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:07.383 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.381+0000 7f1a35b9d700 1 --2- 172.21.15.27:0/246569149 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1a10066f30 0x7f1a100693e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1a280074f0 tx=0x7f1a28009700 comp rx=0 tx=0).stop 2023-10-06T01:42:07.383 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.381+0000 7f1a35b9d700 1 -- 172.21.15.27:0/246569149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a300b0a20 msgr2=0x7f1a300c7450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:07.383 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.381+0000 7f1a35b9d700 1 --2- 172.21.15.27:0/246569149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a300b0a20 0x7f1a300c7450 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1a2000c590 tx=0x7f1a2000c950 comp rx=0 tx=0).stop 2023-10-06T01:42:07.383 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.381+0000 7f1a35b9d700 1 -- 172.21.15.27:0/246569149 shutdown_connections 2023-10-06T01:42:07.383 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.381+0000 7f1a35b9d700 1 --2- 172.21.15.27:0/246569149 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1a10066f30 0x7f1a100693e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:07.384 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.381+0000 7f1a35b9d700 1 --2- 172.21.15.27:0/246569149 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f1a30069da0 0x7f1a30140660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:07.384 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.381+0000 7f1a35b9d700 1 --2- 172.21.15.27:0/246569149 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1a3015d480 0x7f1a300cadd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:07.384 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.381+0000 7f1a35b9d700 1 --2- 172.21.15.27:0/246569149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1a300b0a20 0x7f1a300c7450 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:07.384 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.381+0000 7f1a35b9d700 1 -- 172.21.15.27:0/246569149 >> 172.21.15.27:0/246569149 conn(0x7f1a300ccf80 msgr2=0x7f1a300cf9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:07.384 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.381+0000 7f1a35b9d700 1 -- 172.21.15.27:0/246569149 shutdown_connections 2023-10-06T01:42:07.384 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.381+0000 7f1a35b9d700 1 -- 172.21.15.27:0/246569149 wait complete. 2023-10-06T01:42:07.395 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-06T01:42:07.395 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-06T01:42:07.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.512+0000 7f141c24c700 1 Processor -- start 2023-10-06T01:42:07.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.513+0000 7f141c24c700 1 -- start start 2023-10-06T01:42:07.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.514+0000 7f141c24c700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f141415d560 0x7f14140d0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:07.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.514+0000 7f141c24c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f14141591b0 0x7f1414159600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:07.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.514+0000 7f141c24c700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14140b2ad0 con 0x7f141415d930 2023-10-06T01:42:07.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.514+0000 7f141c24c700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14140b4160 con 0x7f14141591b0 2023-10-06T01:42:07.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.514+0000 7f141c24c700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14140b2230 con 0x7f141415d560 2023-10-06T01:42:07.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.514+0000 7f141a7e9700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f14141591b0 0x7f1414159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:07.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.514+0000 7f141a7e9700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f14141591b0 0x7f1414159600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37432/0 (socket says 172.21.15.27:37432) 2023-10-06T01:42:07.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.514+0000 7f141a7e9700 1 -- 172.21.15.27:0/3966560119 learned_addr learned my addr 172.21.15.27:0/3966560119 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:07.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.514+0000 7f1419fe8700 1 --2- 172.21.15.27:0/3966560119 >> v2:172.21.15.124:3300/0 conn(0x7f141415d560 0x7f14140d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:07.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.514+0000 7f141a7e9700 1 -- 172.21.15.27:0/3966560119 >> v2:172.21.15.124:3300/0 conn(0x7f141415d560 msgr2=0x7f14140d0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:07.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.514+0000 7f141a7e9700 1 --2- 172.21.15.27:0/3966560119 >> v2:172.21.15.124:3300/0 conn(0x7f141415d560 0x7f14140d0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.514+0000 7f141a7e9700 1 -- 172.21.15.27:0/3966560119 >> v1:172.21.15.124:6789/0 conn(0x7f141415d930 legacy=0x7f14140b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.514+0000 7f141a7e9700 1 -- 172.21.15.27:0/3966560119 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14140b2510 con 0x7f14141591b0 2023-10-06T01:42:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.514+0000 7f141a7e9700 1 --2- 172.21.15.27:0/3966560119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14141591b0 0x7f1414159600 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1414071410 tx=0x7f1404009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38c9a6841a806bb4 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.514+0000 7f1418fe6700 1 -- 172.21.15.27:0/3966560119 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f140400e070 con 0x7f14141591b0 2023-10-06T01:42:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.515+0000 7f1418fe6700 1 -- 172.21.15.27:0/3966560119 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f140400ba50 con 0x7f14141591b0 2023-10-06T01:42:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.515+0000 7f1418fe6700 1 -- 172.21.15.27:0/3966560119 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14040092e0 con 0x7f14141591b0 2023-10-06T01:42:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.515+0000 7f141c24c700 1 -- 172.21.15.27:0/3966560119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14141591b0 msgr2=0x7f1414159600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.515+0000 7f141c24c700 1 --2- 172.21.15.27:0/3966560119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14141591b0 0x7f1414159600 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1414071410 tx=0x7f1404009840 comp rx=0 tx=0).stop 2023-10-06T01:42:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.515+0000 7f141c24c700 1 -- 172.21.15.27:0/3966560119 shutdown_connections 2023-10-06T01:42:07.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.515+0000 7f141c24c700 1 --2- 172.21.15.27:0/3966560119 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14141591b0 0x7f1414159600 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:07.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.515+0000 7f141c24c700 1 --2- 172.21.15.27:0/3966560119 >> v2:172.21.15.124:3300/0 conn(0x7f141415d560 0x7f14140d0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:07.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.515+0000 7f141c24c700 1 -- 172.21.15.27:0/3966560119 >> 172.21.15.27:0/3966560119 conn(0x7f14140d1180 msgr2=0x7f1414069640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:07.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.515+0000 7f141c24c700 1 -- 172.21.15.27:0/3966560119 shutdown_connections 2023-10-06T01:42:07.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.515+0000 7f141c24c700 1 -- 172.21.15.27:0/3966560119 wait complete. 2023-10-06T01:42:07.521 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.516+0000 7f141c24c700 1 Processor -- start 2023-10-06T01:42:07.521 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.516+0000 7f141c24c700 1 -- start start 2023-10-06T01:42:07.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.517+0000 7f141c24c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14141591b0 0x7f1414147260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:07.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.517+0000 7f141c24c700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f141415d560 0x7f14140b0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:07.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.517+0000 7f141c24c700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f141415d930 0x7f14140b1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:07.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.517+0000 7f141c24c700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14141477a0 con 0x7f14141591b0 2023-10-06T01:42:07.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.517+0000 7f141c24c700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1414159af0 con 0x7f141415d930 2023-10-06T01:42:07.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.517+0000 7f141c24c700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1414159dd0 con 0x7f141415d560 2023-10-06T01:42:07.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.517+0000 7f1419fe8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14141591b0 0x7f1414147260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:07.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.517+0000 7f1419fe8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14141591b0 0x7f1414147260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37440/0 (socket says 172.21.15.27:37440) 2023-10-06T01:42:07.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.517+0000 7f1419fe8700 1 -- 172.21.15.27:0/3507440341 learned_addr learned my addr 172.21.15.27:0/3507440341 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:07.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.517+0000 7f14197e7700 1 --2- 172.21.15.27:0/3507440341 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f141415d560 0x7f14140b0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:07.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.517+0000 7f1419fe8700 1 -- 172.21.15.27:0/3507440341 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f141415d560 msgr2=0x7f14140b0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:07.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.517+0000 7f1419fe8700 1 --2- 172.21.15.27:0/3507440341 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f141415d560 0x7f14140b0b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:07.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.517+0000 7f141a7e9700 1 --2- 172.21.15.27:0/3507440341 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f141415d930 0x7f14140b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:07.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.517+0000 7f1419fe8700 1 -- 172.21.15.27:0/3507440341 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f141415d930 msgr2=0x7f14140b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:07.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.517+0000 7f1419fe8700 1 --2- 172.21.15.27:0/3507440341 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f141415d930 0x7f14140b1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:07.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.517+0000 7f1419fe8700 1 -- 172.21.15.27:0/3507440341 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1404009ae0 con 0x7f14141591b0 2023-10-06T01:42:07.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.518+0000 7f1419fe8700 1 --2- 172.21.15.27:0/3507440341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14141591b0 0x7f1414147260 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f140c00d890 tx=0x7f140c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:07.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.518+0000 7f1411ffb700 1 -- 172.21.15.27:0/3507440341 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f140c00b600 con 0x7f14141591b0 2023-10-06T01:42:07.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.518+0000 7f1411ffb700 1 -- 172.21.15.27:0/3507440341 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f140c00b760 con 0x7f14141591b0 2023-10-06T01:42:07.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.518+0000 7f141c24c700 1 -- 172.21.15.27:0/3507440341 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f141415a0b0 con 0x7f14141591b0 2023-10-06T01:42:07.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.518+0000 7f1411ffb700 1 -- 172.21.15.27:0/3507440341 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f140c00bae0 con 0x7f14141591b0 2023-10-06T01:42:07.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.518+0000 7f141c24c700 1 -- 172.21.15.27:0/3507440341 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1414154b70 con 0x7f14141591b0 2023-10-06T01:42:07.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.519+0000 7f141c24c700 1 -- 172.21.15.27:0/3507440341 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1414064050 con 0x7f14141591b0 2023-10-06T01:42:07.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.519+0000 7f1411ffb700 1 -- 172.21.15.27:0/3507440341 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f140c01e460 con 0x7f14141591b0 2023-10-06T01:42:07.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.520+0000 7f1411ffb700 1 --2- 172.21.15.27:0/3507440341 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1400066f30 0x7f14000693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:07.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.520+0000 7f1411ffb700 1 -- 172.21.15.27:0/3507440341 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f140c084d70 con 0x7f14141591b0 2023-10-06T01:42:07.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.520+0000 7f14197e7700 1 --2- 172.21.15.27:0/3507440341 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1400066f30 0x7f14000693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:07.525 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.520+0000 7f14197e7700 1 --2- 172.21.15.27:0/3507440341 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1400066f30 0x7f14000693e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1414157e40 tx=0x7f1408009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:07.525 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.524+0000 7f1411ffb700 1 -- 172.21.15.27:0/3507440341 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f140c051e80 con 0x7f14141591b0 2023-10-06T01:42:07.616 INFO:tasks.ceph.mds.d.smithi124.stderr:2023-10-06T01:42:07.616+0000 7f17014a6700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-06T01:42:07.716 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.714+0000 7f141c24c700 1 -- 172.21.15.27:0/3507440341 --> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f141415ae70 con 0x7f1400066f30 2023-10-06T01:42:07.724 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.723+0000 7f1411ffb700 1 -- 172.21.15.27:0/3507440341 <== mgr.4100 v2:172.21.15.124:6832/90278 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+277611 (secure 0 0 0) 0x7f141415ae70 con 0x7f1400066f30 2023-10-06T01:42:07.728 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:07.730 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.726+0000 7f141c24c700 1 -- 172.21.15.27:0/3507440341 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1400066f30 msgr2=0x7f14000693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:07.730 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.726+0000 7f141c24c700 1 --2- 172.21.15.27:0/3507440341 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1400066f30 0x7f14000693e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1414157e40 tx=0x7f1408009040 comp rx=0 tx=0).stop 2023-10-06T01:42:07.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.726+0000 7f141c24c700 1 -- 172.21.15.27:0/3507440341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14141591b0 msgr2=0x7f1414147260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:07.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.726+0000 7f141c24c700 1 --2- 172.21.15.27:0/3507440341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14141591b0 0x7f1414147260 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f140c00d890 tx=0x7f140c008040 comp rx=0 tx=0).stop 2023-10-06T01:42:07.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.726+0000 7f141c24c700 1 -- 172.21.15.27:0/3507440341 shutdown_connections 2023-10-06T01:42:07.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.726+0000 7f141c24c700 1 --2- 172.21.15.27:0/3507440341 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1400066f30 0x7f14000693e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:07.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.726+0000 7f141c24c700 1 --2- 172.21.15.27:0/3507440341 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f141415d930 0x7f14140b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:07.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.726+0000 7f141c24c700 1 --2- 172.21.15.27:0/3507440341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f14141591b0 0x7f1414147260 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:07.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.726+0000 7f141c24c700 1 --2- 172.21.15.27:0/3507440341 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f141415d560 0x7f14140b0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:07.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.726+0000 7f141c24c700 1 -- 172.21.15.27:0/3507440341 >> 172.21.15.27:0/3507440341 conn(0x7f14140d1180 msgr2=0x7f14141580b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:07.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.726+0000 7f141c24c700 1 -- 172.21.15.27:0/3507440341 shutdown_connections 2023-10-06T01:42:07.732 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.726+0000 7f141c24c700 1 -- 172.21.15.27:0/3507440341 wait complete. 2023-10-06T01:42:07.732 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-06T01:42:07.740 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":54,"stamp":"2023-10-06T01:42:06.050885+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591938,"num_objects":9,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":171,"num_write_kb":2425,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":88,"acting":88,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2381916,"kb_used_data":3612,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752592804,"statfs":{"total":773094113280,"available":770655031296,"internally_reserved":0,"allocated":3698688,"data_stored":2153520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1570,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.424679"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":8,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.658560+0000","last_change":"2023-10-06T01:42:03.658560+0000","last_active":"2023-10-06T01:42:03.658560+0000","last_peered":"2023-10-06T01:42:03.658560+0000","last_clean":"2023-10-06T01:42:03.658560+0000","last_became_active":"2023-10-06T01:42:03.658288+0000","last_became_peered":"2023-10-06T01:42:03.658288+0000","last_unstale":"2023-10-06T01:42:03.658560+0000","last_undegraded":"2023-10-06T01:42:03.658560+0000","last_fullsized":"2023-10-06T01:42:03.658560+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:43:09.292191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":8,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.649498+0000","last_change":"2023-10-06T01:42:03.649498+0000","last_active":"2023-10-06T01:42:03.649498+0000","last_peered":"2023-10-06T01:42:03.649498+0000","last_clean":"2023-10-06T01:42:03.649498+0000","last_became_active":"2023-10-06T01:42:03.649252+0000","last_became_peered":"2023-10-06T01:42:03.649252+0000","last_unstale":"2023-10-06T01:42:03.649498+0000","last_undegraded":"2023-10-06T01:42:03.649498+0000","last_fullsized":"2023-10-06T01:42:03.649498+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:17:06.578187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.662639+0000","last_change":"2023-10-06T01:42:03.655756+0000","last_active":"2023-10-06T01:42:03.662639+0000","last_peered":"2023-10-06T01:42:03.662639+0000","last_clean":"2023-10-06T01:42:03.662639+0000","last_became_active":"2023-10-06T01:42:03.655266+0000","last_became_peered":"2023-10-06T01:42:03.655266+0000","last_unstale":"2023-10-06T01:42:03.662639+0000","last_undegraded":"2023-10-06T01:42:03.662639+0000","last_fullsized":"2023-10-06T01:42:03.662639+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:34:59.205863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":8,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.652127+0000","last_change":"2023-10-06T01:42:03.652127+0000","last_active":"2023-10-06T01:42:03.652127+0000","last_peered":"2023-10-06T01:42:03.652127+0000","last_clean":"2023-10-06T01:42:03.652127+0000","last_became_active":"2023-10-06T01:42:03.651874+0000","last_became_peered":"2023-10-06T01:42:03.651874+0000","last_unstale":"2023-10-06T01:42:03.652127+0000","last_undegraded":"2023-10-06T01:42:03.652127+0000","last_fullsized":"2023-10-06T01:42:03.652127+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:27:51.973115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":8,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.653994+0000","last_change":"2023-10-06T01:42:03.653994+0000","last_active":"2023-10-06T01:42:03.653994+0000","last_peered":"2023-10-06T01:42:03.653994+0000","last_clean":"2023-10-06T01:42:03.653994+0000","last_became_active":"2023-10-06T01:42:03.653698+0000","last_became_peered":"2023-10-06T01:42:03.653698+0000","last_unstale":"2023-10-06T01:42:03.653994+0000","last_undegraded":"2023-10-06T01:42:03.653994+0000","last_fullsized":"2023-10-06T01:42:03.653994+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:18:14.223068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.662622+0000","last_change":"2023-10-06T01:42:03.655576+0000","last_active":"2023-10-06T01:42:03.662622+0000","last_peered":"2023-10-06T01:42:03.662622+0000","last_clean":"2023-10-06T01:42:03.662622+0000","last_became_active":"2023-10-06T01:42:03.655228+0000","last_became_peered":"2023-10-06T01:42:03.655228+0000","last_unstale":"2023-10-06T01:42:03.662622+0000","last_undegraded":"2023-10-06T01:42:03.662622+0000","last_fullsized":"2023-10-06T01:42:03.662622+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:17:06.578187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.662381+0000","last_change":"2023-10-06T01:42:03.648412+0000","last_active":"2023-10-06T01:42:03.662381+0000","last_peered":"2023-10-06T01:42:03.662381+0000","last_clean":"2023-10-06T01:42:03.662381+0000","last_became_active":"2023-10-06T01:42:03.648159+0000","last_became_peered":"2023-10-06T01:42:03.648159+0000","last_unstale":"2023-10-06T01:42:03.662381+0000","last_undegraded":"2023-10-06T01:42:03.662381+0000","last_fullsized":"2023-10-06T01:42:03.662381+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:47:42.520312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":8,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.655189+0000","last_change":"2023-10-06T01:42:03.655189+0000","last_active":"2023-10-06T01:42:03.655189+0000","last_peered":"2023-10-06T01:42:03.655189+0000","last_clean":"2023-10-06T01:42:03.655189+0000","last_became_active":"2023-10-06T01:42:03.654961+0000","last_became_peered":"2023-10-06T01:42:03.654961+0000","last_unstale":"2023-10-06T01:42:03.655189+0000","last_undegraded":"2023-10-06T01:42:03.655189+0000","last_fullsized":"2023-10-06T01:42:03.655189+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:56:53.849323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":8,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.647572+0000","last_change":"2023-10-06T01:42:03.647572+0000","last_active":"2023-10-06T01:42:03.647572+0000","last_peered":"2023-10-06T01:42:03.647572+0000","last_clean":"2023-10-06T01:42:03.647572+0000","last_became_active":"2023-10-06T01:42:03.647288+0000","last_became_peered":"2023-10-06T01:42:03.647288+0000","last_unstale":"2023-10-06T01:42:03.647572+0000","last_undegraded":"2023-10-06T01:42:03.647572+0000","last_fullsized":"2023-10-06T01:42:03.647572+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:06:42.668497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:01.619802+0000","last_change":"2023-10-06T01:42:01.619802+0000","last_active":"2023-10-06T01:42:01.619802+0000","last_peered":"2023-10-06T01:42:01.619802+0000","last_clean":"2023-10-06T01:42:01.619802+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:01.619802+0000","last_undegraded":"2023-10-06T01:42:01.619802+0000","last_fullsized":"2023-10-06T01:42:01.619802+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":8,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.654697+0000","last_change":"2023-10-06T01:42:03.654697+0000","last_active":"2023-10-06T01:42:03.654697+0000","last_peered":"2023-10-06T01:42:03.654697+0000","last_clean":"2023-10-06T01:42:03.654697+0000","last_became_active":"2023-10-06T01:42:03.654384+0000","last_became_peered":"2023-10-06T01:42:03.654384+0000","last_unstale":"2023-10-06T01:42:03.654697+0000","last_undegraded":"2023-10-06T01:42:03.654697+0000","last_fullsized":"2023-10-06T01:42:03.654697+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:10:24.478901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"30'1","reported_seq":11,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.655960+0000","last_change":"2023-10-06T01:42:02.646451+0000","last_active":"2023-10-06T01:42:03.655960+0000","last_peered":"2023-10-06T01:42:03.655960+0000","last_clean":"2023-10-06T01:42:03.655960+0000","last_became_active":"2023-10-06T01:42:02.646182+0000","last_became_peered":"2023-10-06T01:42:02.646182+0000","last_unstale":"2023-10-06T01:42:03.655960+0000","last_undegraded":"2023-10-06T01:42:03.655960+0000","last_fullsized":"2023-10-06T01:42:03.655960+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:20:15.728651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1d","version":"30'2","reported_seq":11,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.660139+0000","last_change":"2023-10-06T01:42:02.649210+0000","last_active":"2023-10-06T01:42:03.660139+0000","last_peered":"2023-10-06T01:42:03.660139+0000","last_clean":"2023-10-06T01:42:03.660139+0000","last_became_active":"2023-10-06T01:42:02.648923+0000","last_became_peered":"2023-10-06T01:42:02.648923+0000","last_unstale":"2023-10-06T01:42:03.660139+0000","last_undegraded":"2023-10-06T01:42:03.660139+0000","last_fullsized":"2023-10-06T01:42:03.660139+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:02:21.168465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":8,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.653802+0000","last_change":"2023-10-06T01:42:03.653802+0000","last_active":"2023-10-06T01:42:03.653802+0000","last_peered":"2023-10-06T01:42:03.653802+0000","last_clean":"2023-10-06T01:42:03.653802+0000","last_became_active":"2023-10-06T01:42:03.653504+0000","last_became_peered":"2023-10-06T01:42:03.653504+0000","last_unstale":"2023-10-06T01:42:03.653802+0000","last_undegraded":"2023-10-06T01:42:03.653802+0000","last_fullsized":"2023-10-06T01:42:03.653802+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:56:10.182021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:01.619802+0000","last_change":"2023-10-06T01:42:01.619802+0000","last_active":"2023-10-06T01:42:01.619802+0000","last_peered":"2023-10-06T01:42:01.619802+0000","last_clean":"2023-10-06T01:42:01.619802+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:01.619802+0000","last_undegraded":"2023-10-06T01:42:01.619802+0000","last_fullsized":"2023-10-06T01:42:01.619802+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.646780+0000","last_change":"2023-10-06T01:42:03.641991+0000","last_active":"2023-10-06T01:42:03.646780+0000","last_peered":"2023-10-06T01:42:03.646780+0000","last_clean":"2023-10-06T01:42:03.646780+0000","last_became_active":"2023-10-06T01:42:03.641759+0000","last_became_peered":"2023-10-06T01:42:03.641759+0000","last_unstale":"2023-10-06T01:42:03.646780+0000","last_undegraded":"2023-10-06T01:42:03.646780+0000","last_fullsized":"2023-10-06T01:42:03.646780+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:06:21.581530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:01.619802+0000","last_change":"2023-10-06T01:42:01.619802+0000","last_active":"2023-10-06T01:42:01.619802+0000","last_peered":"2023-10-06T01:42:01.619802+0000","last_clean":"2023-10-06T01:42:01.619802+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:01.619802+0000","last_undegraded":"2023-10-06T01:42:01.619802+0000","last_fullsized":"2023-10-06T01:42:01.619802+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:01.619802+0000","last_change":"2023-10-06T01:42:01.619802+0000","last_active":"2023-10-06T01:42:01.619802+0000","last_peered":"2023-10-06T01:42:01.619802+0000","last_clean":"2023-10-06T01:42:01.619802+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:01.619802+0000","last_undegraded":"2023-10-06T01:42:01.619802+0000","last_fullsized":"2023-10-06T01:42:01.619802+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-06T01:42:02.654661+0000","last_change":"2023-10-06T01:42:02.651745+0000","last_active":"2023-10-06T01:42:02.654661+0000","last_peered":"2023-10-06T01:42:02.654661+0000","last_clean":"2023-10-06T01:42:02.654661+0000","last_became_active":"2023-10-06T01:42:02.651455+0000","last_became_peered":"2023-10-06T01:42:02.651455+0000","last_unstale":"2023-10-06T01:42:02.654661+0000","last_undegraded":"2023-10-06T01:42:02.654661+0000","last_fullsized":"2023-10-06T01:42:02.654661+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:36:37.018597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":8,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.656442+0000","last_change":"2023-10-06T01:42:03.656442+0000","last_active":"2023-10-06T01:42:03.656442+0000","last_peered":"2023-10-06T01:42:03.656442+0000","last_clean":"2023-10-06T01:42:03.656442+0000","last_became_active":"2023-10-06T01:42:03.656218+0000","last_became_peered":"2023-10-06T01:42:03.656218+0000","last_unstale":"2023-10-06T01:42:03.656442+0000","last_undegraded":"2023-10-06T01:42:03.656442+0000","last_fullsized":"2023-10-06T01:42:03.656442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:16:29.730704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:01.619802+0000","last_change":"2023-10-06T01:42:01.619802+0000","last_active":"2023-10-06T01:42:01.619802+0000","last_peered":"2023-10-06T01:42:01.619802+0000","last_clean":"2023-10-06T01:42:01.619802+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:01.619802+0000","last_undegraded":"2023-10-06T01:42:01.619802+0000","last_fullsized":"2023-10-06T01:42:01.619802+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-06T01:42:02.650378+0000","last_change":"2023-10-06T01:42:02.647608+0000","last_active":"2023-10-06T01:42:02.650378+0000","last_peered":"2023-10-06T01:42:02.650378+0000","last_clean":"2023-10-06T01:42:02.650378+0000","last_became_active":"2023-10-06T01:42:02.647350+0000","last_became_peered":"2023-10-06T01:42:02.647350+0000","last_unstale":"2023-10-06T01:42:02.650378+0000","last_undegraded":"2023-10-06T01:42:02.650378+0000","last_fullsized":"2023-10-06T01:42:02.650378+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:27:50.966966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:01.619802+0000","last_change":"2023-10-06T01:42:01.619802+0000","last_active":"2023-10-06T01:42:01.619802+0000","last_peered":"2023-10-06T01:42:01.619802+0000","last_clean":"2023-10-06T01:42:01.619802+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:01.619802+0000","last_undegraded":"2023-10-06T01:42:01.619802+0000","last_fullsized":"2023-10-06T01:42:01.619802+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.662932+0000","last_change":"2023-10-06T01:42:03.658539+0000","last_active":"2023-10-06T01:42:03.662932+0000","last_peered":"2023-10-06T01:42:03.662932+0000","last_clean":"2023-10-06T01:42:03.662932+0000","last_became_active":"2023-10-06T01:42:03.658265+0000","last_became_peered":"2023-10-06T01:42:03.658265+0000","last_unstale":"2023-10-06T01:42:03.662932+0000","last_undegraded":"2023-10-06T01:42:03.662932+0000","last_fullsized":"2023-10-06T01:42:03.662932+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:01:14.549337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:01.619802+0000","last_change":"2023-10-06T01:42:01.619802+0000","last_active":"2023-10-06T01:42:01.619802+0000","last_peered":"2023-10-06T01:42:01.619802+0000","last_clean":"2023-10-06T01:42:01.619802+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:01.619802+0000","last_undegraded":"2023-10-06T01:42:01.619802+0000","last_fullsized":"2023-10-06T01:42:01.619802+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"30'6","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.656461+0000","last_change":"2023-10-06T01:42:02.649160+0000","last_active":"2023-10-06T01:42:03.656461+0000","last_peered":"2023-10-06T01:42:03.656461+0000","last_clean":"2023-10-06T01:42:03.656461+0000","last_became_active":"2023-10-06T01:42:02.648829+0000","last_became_peered":"2023-10-06T01:42:02.648829+0000","last_unstale":"2023-10-06T01:42:03.656461+0000","last_undegraded":"2023-10-06T01:42:03.656461+0000","last_fullsized":"2023-10-06T01:42:03.656461+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:06:41.662348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.662662+0000","last_change":"2023-10-06T01:42:03.655894+0000","last_active":"2023-10-06T01:42:03.662662+0000","last_peered":"2023-10-06T01:42:03.662662+0000","last_clean":"2023-10-06T01:42:03.662662+0000","last_became_active":"2023-10-06T01:42:03.655283+0000","last_became_peered":"2023-10-06T01:42:03.655283+0000","last_unstale":"2023-10-06T01:42:03.662662+0000","last_undegraded":"2023-10-06T01:42:03.662662+0000","last_fullsized":"2023-10-06T01:42:03.662662+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:37:15.263123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"30'2","reported_seq":11,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.658107+0000","last_change":"2023-10-06T01:42:02.642881+0000","last_active":"2023-10-06T01:42:03.658107+0000","last_peered":"2023-10-06T01:42:03.658107+0000","last_clean":"2023-10-06T01:42:03.658107+0000","last_became_active":"2023-10-06T01:42:02.642598+0000","last_became_peered":"2023-10-06T01:42:02.642598+0000","last_unstale":"2023-10-06T01:42:03.658107+0000","last_undegraded":"2023-10-06T01:42:03.658107+0000","last_fullsized":"2023-10-06T01:42:03.658107+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:03:48.160726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":8,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.649487+0000","last_change":"2023-10-06T01:42:03.649487+0000","last_active":"2023-10-06T01:42:03.649487+0000","last_peered":"2023-10-06T01:42:03.649487+0000","last_clean":"2023-10-06T01:42:03.649487+0000","last_became_active":"2023-10-06T01:42:03.649193+0000","last_became_peered":"2023-10-06T01:42:03.649193+0000","last_unstale":"2023-10-06T01:42:03.649487+0000","last_undegraded":"2023-10-06T01:42:03.649487+0000","last_fullsized":"2023-10-06T01:42:03.649487+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:17:15.841349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-06T01:42:02.654678+0000","last_change":"2023-10-06T01:42:02.652300+0000","last_active":"2023-10-06T01:42:02.654678+0000","last_peered":"2023-10-06T01:42:02.654678+0000","last_clean":"2023-10-06T01:42:02.654678+0000","last_became_active":"2023-10-06T01:42:02.652043+0000","last_became_peered":"2023-10-06T01:42:02.652043+0000","last_unstale":"2023-10-06T01:42:02.654678+0000","last_undegraded":"2023-10-06T01:42:02.654678+0000","last_fullsized":"2023-10-06T01:42:02.654678+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:43:36.333908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:01.619802+0000","last_change":"2023-10-06T01:42:01.619802+0000","last_active":"2023-10-06T01:42:01.619802+0000","last_peered":"2023-10-06T01:42:01.619802+0000","last_clean":"2023-10-06T01:42:01.619802+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:01.619802+0000","last_undegraded":"2023-10-06T01:42:01.619802+0000","last_fullsized":"2023-10-06T01:42:01.619802+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":8,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.658901+0000","last_change":"2023-10-06T01:42:03.658901+0000","last_active":"2023-10-06T01:42:03.658901+0000","last_peered":"2023-10-06T01:42:03.658901+0000","last_clean":"2023-10-06T01:42:03.658901+0000","last_became_active":"2023-10-06T01:42:03.658334+0000","last_became_peered":"2023-10-06T01:42:03.658334+0000","last_unstale":"2023-10-06T01:42:03.658901+0000","last_undegraded":"2023-10-06T01:42:03.658901+0000","last_fullsized":"2023-10-06T01:42:03.658901+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:23:52.767666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:01.619802+0000","last_change":"2023-10-06T01:42:01.619802+0000","last_active":"2023-10-06T01:42:01.619802+0000","last_peered":"2023-10-06T01:42:01.619802+0000","last_clean":"2023-10-06T01:42:01.619802+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:01.619802+0000","last_undegraded":"2023-10-06T01:42:01.619802+0000","last_fullsized":"2023-10-06T01:42:01.619802+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":8,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.654313+0000","last_change":"2023-10-06T01:42:03.654313+0000","last_active":"2023-10-06T01:42:03.654313+0000","last_peered":"2023-10-06T01:42:03.654313+0000","last_clean":"2023-10-06T01:42:03.654313+0000","last_became_active":"2023-10-06T01:42:03.654030+0000","last_became_peered":"2023-10-06T01:42:03.654030+0000","last_unstale":"2023-10-06T01:42:03.654313+0000","last_undegraded":"2023-10-06T01:42:03.654313+0000","last_fullsized":"2023-10-06T01:42:03.654313+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:55:56.347224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:01.619802+0000","last_change":"2023-10-06T01:42:01.619802+0000","last_active":"2023-10-06T01:42:01.619802+0000","last_peered":"2023-10-06T01:42:01.619802+0000","last_clean":"2023-10-06T01:42:01.619802+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:01.619802+0000","last_undegraded":"2023-10-06T01:42:01.619802+0000","last_fullsized":"2023-10-06T01:42:01.619802+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:01.619802+0000","last_change":"2023-10-06T01:42:01.619802+0000","last_active":"2023-10-06T01:42:01.619802+0000","last_peered":"2023-10-06T01:42:01.619802+0000","last_clean":"2023-10-06T01:42:01.619802+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:01.619802+0000","last_undegraded":"2023-10-06T01:42:01.619802+0000","last_fullsized":"2023-10-06T01:42:01.619802+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:01.619802+0000","last_change":"2023-10-06T01:42:01.619802+0000","last_active":"2023-10-06T01:42:01.619802+0000","last_peered":"2023-10-06T01:42:01.619802+0000","last_clean":"2023-10-06T01:42:01.619802+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:01.619802+0000","last_undegraded":"2023-10-06T01:42:01.619802+0000","last_fullsized":"2023-10-06T01:42:01.619802+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-06T01:42:02.654149+0000","last_change":"2023-10-06T01:42:02.649181+0000","last_active":"2023-10-06T01:42:02.654149+0000","last_peered":"2023-10-06T01:42:02.654149+0000","last_clean":"2023-10-06T01:42:02.654149+0000","last_became_active":"2023-10-06T01:42:02.648911+0000","last_became_peered":"2023-10-06T01:42:02.648911+0000","last_unstale":"2023-10-06T01:42:02.654149+0000","last_undegraded":"2023-10-06T01:42:02.654149+0000","last_fullsized":"2023-10-06T01:42:02.654149+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:15:03.482499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:01.619802+0000","last_change":"2023-10-06T01:42:01.619802+0000","last_active":"2023-10-06T01:42:01.619802+0000","last_peered":"2023-10-06T01:42:01.619802+0000","last_clean":"2023-10-06T01:42:01.619802+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:01.619802+0000","last_undegraded":"2023-10-06T01:42:01.619802+0000","last_fullsized":"2023-10-06T01:42:01.619802+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.662472+0000","last_change":"2023-10-06T01:42:03.654680+0000","last_active":"2023-10-06T01:42:03.662472+0000","last_peered":"2023-10-06T01:42:03.662472+0000","last_clean":"2023-10-06T01:42:03.662472+0000","last_became_active":"2023-10-06T01:42:03.654382+0000","last_became_peered":"2023-10-06T01:42:03.654382+0000","last_unstale":"2023-10-06T01:42:03.662472+0000","last_undegraded":"2023-10-06T01:42:03.662472+0000","last_fullsized":"2023-10-06T01:42:03.662472+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:50:55.550267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:01.619802+0000","last_change":"2023-10-06T01:42:01.619802+0000","last_active":"2023-10-06T01:42:01.619802+0000","last_peered":"2023-10-06T01:42:01.619802+0000","last_clean":"2023-10-06T01:42:01.619802+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:01.619802+0000","last_undegraded":"2023-10-06T01:42:01.619802+0000","last_fullsized":"2023-10-06T01:42:01.619802+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:01.619802+0000","last_change":"2023-10-06T01:42:01.619802+0000","last_active":"2023-10-06T01:42:01.619802+0000","last_peered":"2023-10-06T01:42:01.619802+0000","last_clean":"2023-10-06T01:42:01.619802+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:01.619802+0000","last_undegraded":"2023-10-06T01:42:01.619802+0000","last_fullsized":"2023-10-06T01:42:01.619802+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":8,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.656556+0000","last_change":"2023-10-06T01:42:03.656556+0000","last_active":"2023-10-06T01:42:03.656556+0000","last_peered":"2023-10-06T01:42:03.656556+0000","last_clean":"2023-10-06T01:42:03.656556+0000","last_became_active":"2023-10-06T01:42:03.656233+0000","last_became_peered":"2023-10-06T01:42:03.656233+0000","last_unstale":"2023-10-06T01:42:03.656556+0000","last_undegraded":"2023-10-06T01:42:03.656556+0000","last_fullsized":"2023-10-06T01:42:03.656556+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:56:54.630363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:01.619802+0000","last_change":"2023-10-06T01:42:01.619802+0000","last_active":"2023-10-06T01:42:01.619802+0000","last_peered":"2023-10-06T01:42:01.619802+0000","last_clean":"2023-10-06T01:42:01.619802+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:01.619802+0000","last_undegraded":"2023-10-06T01:42:01.619802+0000","last_fullsized":"2023-10-06T01:42:01.619802+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:01.619802+0000","last_change":"2023-10-06T01:42:01.619802+0000","last_active":"2023-10-06T01:42:01.619802+0000","last_peered":"2023-10-06T01:42:01.619802+0000","last_clean":"2023-10-06T01:42:01.619802+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:01.619802+0000","last_undegraded":"2023-10-06T01:42:01.619802+0000","last_fullsized":"2023-10-06T01:42:01.619802+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":19,"reported_epoch":25,"state":"active+clean","last_fresh":"2023-10-06T01:41:56.606871+0000","last_change":"2023-10-06T01:41:07.454295+0000","last_active":"2023-10-06T01:41:56.606871+0000","last_peered":"2023-10-06T01:41:56.606871+0000","last_clean":"2023-10-06T01:41:56.606871+0000","last_became_active":"2023-10-06T01:41:07.454034+0000","last_became_peered":"2023-10-06T01:41:07.454034+0000","last_unstale":"2023-10-06T01:41:56.606871+0000","last_undegraded":"2023-10-06T01:41:56.606871+0000","last_fullsized":"2023-10-06T01:41:56.606871+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:32:58.914183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-06T01:42:02.648150+0000","last_change":"2023-10-06T01:42:02.644525+0000","last_active":"2023-10-06T01:42:02.648150+0000","last_peered":"2023-10-06T01:42:02.648150+0000","last_clean":"2023-10-06T01:42:02.648150+0000","last_became_active":"2023-10-06T01:42:02.644226+0000","last_became_peered":"2023-10-06T01:42:02.644226+0000","last_unstale":"2023-10-06T01:42:02.648150+0000","last_undegraded":"2023-10-06T01:42:02.648150+0000","last_fullsized":"2023-10-06T01:42:02.648150+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:40:35.423082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-06T01:42:02.632076+0000","last_change":"2023-10-06T01:41:07.261955+0000","last_active":"2023-10-06T01:42:02.632076+0000","last_peered":"2023-10-06T01:42:02.632076+0000","last_clean":"2023-10-06T01:42:02.632076+0000","last_became_active":"2023-10-06T01:41:07.261692+0000","last_became_peered":"2023-10-06T01:41:07.261692+0000","last_unstale":"2023-10-06T01:42:02.632076+0000","last_undegraded":"2023-10-06T01:42:02.632076+0000","last_fullsized":"2023-10-06T01:42:02.632076+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":8,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.654638+0000","last_change":"2023-10-06T01:42:03.654638+0000","last_active":"2023-10-06T01:42:03.654638+0000","last_peered":"2023-10-06T01:42:03.654638+0000","last_clean":"2023-10-06T01:42:03.654638+0000","last_became_active":"2023-10-06T01:42:03.654329+0000","last_became_peered":"2023-10-06T01:42:03.654329+0000","last_unstale":"2023-10-06T01:42:03.654638+0000","last_undegraded":"2023-10-06T01:42:03.654638+0000","last_fullsized":"2023-10-06T01:42:03.654638+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:07:16.583330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:01.619802+0000","last_change":"2023-10-06T01:42:01.619802+0000","last_active":"2023-10-06T01:42:01.619802+0000","last_peered":"2023-10-06T01:42:01.619802+0000","last_clean":"2023-10-06T01:42:01.619802+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:01.619802+0000","last_undegraded":"2023-10-06T01:42:01.619802+0000","last_fullsized":"2023-10-06T01:42:01.619802+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-06T01:42:02.643188+0000","last_change":"2023-10-06T01:41:07.258362+0000","last_active":"2023-10-06T01:42:02.643188+0000","last_peered":"2023-10-06T01:42:02.643188+0000","last_clean":"2023-10-06T01:42:02.643188+0000","last_became_active":"2023-10-06T01:41:07.258055+0000","last_became_peered":"2023-10-06T01:41:07.258055+0000","last_unstale":"2023-10-06T01:42:02.643188+0000","last_undegraded":"2023-10-06T01:42:02.643188+0000","last_fullsized":"2023-10-06T01:42:02.643188+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"16'117","reported_seq":167,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-06T01:41:55.908557+0000","last_change":"2023-10-06T01:41:07.261164+0000","last_active":"2023-10-06T01:41:55.908557+0000","last_peered":"2023-10-06T01:41:55.908557+0000","last_clean":"2023-10-06T01:41:55.908557+0000","last_became_active":"2023-10-06T01:41:07.260896+0000","last_became_peered":"2023-10-06T01:41:07.260896+0000","last_unstale":"2023-10-06T01:41:55.908557+0000","last_undegraded":"2023-10-06T01:41:55.908557+0000","last_fullsized":"2023-10-06T01:41:55.908557+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:05:22.934401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:01.619802+0000","last_change":"2023-10-06T01:42:01.619802+0000","last_active":"2023-10-06T01:42:01.619802+0000","last_peered":"2023-10-06T01:42:01.619802+0000","last_clean":"2023-10-06T01:42:01.619802+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:01.619802+0000","last_undegraded":"2023-10-06T01:42:01.619802+0000","last_fullsized":"2023-10-06T01:42:01.619802+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-06T01:41:55.909979+0000","last_change":"2023-10-06T01:41:07.456511+0000","last_active":"2023-10-06T01:41:55.909979+0000","last_peered":"2023-10-06T01:41:55.909979+0000","last_clean":"2023-10-06T01:41:55.909979+0000","last_became_active":"2023-10-06T01:41:07.456024+0000","last_became_peered":"2023-10-06T01:41:07.456024+0000","last_unstale":"2023-10-06T01:41:55.909979+0000","last_undegraded":"2023-10-06T01:41:55.909979+0000","last_fullsized":"2023-10-06T01:41:55.909979+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:51:02.949231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:02.625951+0000","last_change":"2023-10-06T01:42:02.625951+0000","last_active":"2023-10-06T01:42:02.625951+0000","last_peered":"2023-10-06T01:42:02.625951+0000","last_clean":"2023-10-06T01:42:02.625951+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:02.625951+0000","last_undegraded":"2023-10-06T01:42:02.625951+0000","last_fullsized":"2023-10-06T01:42:02.625951+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:01.619802+0000","last_change":"2023-10-06T01:42:01.619802+0000","last_active":"2023-10-06T01:42:01.619802+0000","last_peered":"2023-10-06T01:42:01.619802+0000","last_clean":"2023-10-06T01:42:01.619802+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:01.619802+0000","last_undegraded":"2023-10-06T01:42:01.619802+0000","last_fullsized":"2023-10-06T01:42:01.619802+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":17,"reported_epoch":23,"state":"active+clean","last_fresh":"2023-10-06T01:41:55.908425+0000","last_change":"2023-10-06T01:41:07.456273+0000","last_active":"2023-10-06T01:41:55.908425+0000","last_peered":"2023-10-06T01:41:55.908425+0000","last_clean":"2023-10-06T01:41:55.908425+0000","last_became_active":"2023-10-06T01:41:07.456000+0000","last_became_peered":"2023-10-06T01:41:07.456000+0000","last_unstale":"2023-10-06T01:41:55.908425+0000","last_undegraded":"2023-10-06T01:41:55.908425+0000","last_fullsized":"2023-10-06T01:41:55.908425+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:01:42.633388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":8,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.663945+0000","last_change":"2023-10-06T01:42:03.663945+0000","last_active":"2023-10-06T01:42:03.663945+0000","last_peered":"2023-10-06T01:42:03.663945+0000","last_clean":"2023-10-06T01:42:03.663945+0000","last_became_active":"2023-10-06T01:42:03.663698+0000","last_became_peered":"2023-10-06T01:42:03.663698+0000","last_unstale":"2023-10-06T01:42:03.663945+0000","last_undegraded":"2023-10-06T01:42:03.663945+0000","last_fullsized":"2023-10-06T01:42:03.663945+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:34:59.205863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.636149+0000","last_change":"2023-10-06T01:42:02.650860+0000","last_active":"2023-10-06T01:42:03.636149+0000","last_peered":"2023-10-06T01:42:03.636149+0000","last_clean":"2023-10-06T01:42:03.636149+0000","last_became_active":"2023-10-06T01:42:02.650567+0000","last_became_peered":"2023-10-06T01:42:02.650567+0000","last_unstale":"2023-10-06T01:42:03.636149+0000","last_undegraded":"2023-10-06T01:42:03.636149+0000","last_fullsized":"2023-10-06T01:42:03.636149+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:48:55.898734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":18,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-06T01:41:56.618454+0000","last_change":"2023-10-06T01:41:07.456688+0000","last_active":"2023-10-06T01:41:56.618454+0000","last_peered":"2023-10-06T01:41:56.618454+0000","last_clean":"2023-10-06T01:41:56.618454+0000","last_became_active":"2023-10-06T01:41:07.456304+0000","last_became_peered":"2023-10-06T01:41:07.456304+0000","last_unstale":"2023-10-06T01:41:56.618454+0000","last_undegraded":"2023-10-06T01:41:56.618454+0000","last_fullsized":"2023-10-06T01:41:56.618454+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":8,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.641989+0000","last_change":"2023-10-06T01:42:03.641989+0000","last_active":"2023-10-06T01:42:03.641989+0000","last_peered":"2023-10-06T01:42:03.641989+0000","last_clean":"2023-10-06T01:42:03.641989+0000","last_became_active":"2023-10-06T01:42:03.641714+0000","last_became_peered":"2023-10-06T01:42:03.641714+0000","last_unstale":"2023-10-06T01:42:03.641989+0000","last_undegraded":"2023-10-06T01:42:03.641989+0000","last_fullsized":"2023-10-06T01:42:03.641989+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:24:18.930221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2023-10-06T01:42:01.619802+0000","last_change":"2023-10-06T01:42:01.619802+0000","last_active":"2023-10-06T01:42:01.619802+0000","last_peered":"2023-10-06T01:42:01.619802+0000","last_clean":"2023-10-06T01:42:01.619802+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-06T01:42:01.619802+0000","last_undegraded":"2023-10-06T01:42:01.619802+0000","last_fullsized":"2023-10-06T01:42:01.619802+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-06T01:41:56.616014+0000","last_change":"2023-10-06T01:41:07.456388+0000","last_active":"2023-10-06T01:41:56.616014+0000","last_peered":"2023-10-06T01:41:56.616014+0000","last_clean":"2023-10-06T01:41:56.616014+0000","last_became_active":"2023-10-06T01:41:07.456045+0000","last_became_peered":"2023-10-06T01:41:07.456045+0000","last_unstale":"2023-10-06T01:41:56.616014+0000","last_undegraded":"2023-10-06T01:41:56.616014+0000","last_fullsized":"2023-10-06T01:41:56.616014+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.662503+0000","last_change":"2023-10-06T01:42:03.650881+0000","last_active":"2023-10-06T01:42:03.662503+0000","last_peered":"2023-10-06T01:42:03.662503+0000","last_clean":"2023-10-06T01:42:03.662503+0000","last_became_active":"2023-10-06T01:42:03.650592+0000","last_became_peered":"2023-10-06T01:42:03.650592+0000","last_unstale":"2023-10-06T01:42:03.662503+0000","last_undegraded":"2023-10-06T01:42:03.662503+0000","last_fullsized":"2023-10-06T01:42:03.662503+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:18:14.223068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.635993+0000","last_change":"2023-10-06T01:42:02.650163+0000","last_active":"2023-10-06T01:42:03.635993+0000","last_peered":"2023-10-06T01:42:03.635993+0000","last_clean":"2023-10-06T01:42:03.635993+0000","last_became_active":"2023-10-06T01:42:02.649353+0000","last_became_peered":"2023-10-06T01:42:02.649353+0000","last_unstale":"2023-10-06T01:42:03.635993+0000","last_undegraded":"2023-10-06T01:42:03.635993+0000","last_fullsized":"2023-10-06T01:42:03.635993+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:04:20.088699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":19,"reported_epoch":26,"state":"active+clean","last_fresh":"2023-10-06T01:41:56.616032+0000","last_change":"2023-10-06T01:41:07.258162+0000","last_active":"2023-10-06T01:41:56.616032+0000","last_peered":"2023-10-06T01:41:56.616032+0000","last_clean":"2023-10-06T01:41:56.616032+0000","last_became_active":"2023-10-06T01:41:07.257795+0000","last_became_peered":"2023-10-06T01:41:07.257795+0000","last_unstale":"2023-10-06T01:41:56.616032+0000","last_undegraded":"2023-10-06T01:41:56.616032+0000","last_fullsized":"2023-10-06T01:41:56.616032+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:05:22.934401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":3},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":1570,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":22,"acting":22,"num_store_stats":3},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574865,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298160,"kb_used_data":872,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073680,"statfs":{"total":96636764160,"available":96331448320,"internally_reserved":0,"allocated":892928,"data_stored":704972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38800000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574865,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297576,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074264,"statfs":{"total":96636764160,"available":96332046336,"internally_reserved":0,"allocated":294912,"data_stored":113251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574864,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298200,"kb_used_data":912,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073640,"statfs":{"total":96636764160,"available":96331407360,"internally_reserved":0,"allocated":933888,"data_stored":724952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574864,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297616,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074224,"statfs":{"total":96636764160,"available":96332005376,"internally_reserved":0,"allocated":335872,"data_stored":133175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297580,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074260,"statfs":{"total":96636764160,"available":96332042240,"internally_reserved":0,"allocated":299008,"data_stored":114604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.439}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74199999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71599999999999997}]}]},{"osd":3,"up_from":13,"seq":55834574864,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297628,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074212,"statfs":{"total":96636764160,"available":96331993088,"internally_reserved":0,"allocated":348160,"data_stored":134711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574864,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297580,"kb_used_data":292,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074260,"statfs":{"total":96636764160,"available":96332042240,"internally_reserved":0,"allocated":299008,"data_stored":114604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574862,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297576,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074264,"statfs":{"total":96636764160,"available":96332046336,"internally_reserved":0,"allocated":294912,"data_stored":113251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-06T01:42:07.746 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-06T01:42:07.898 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.897+0000 7fb16a7ed700 1 Processor -- start 2023-10-06T01:42:07.898 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.897+0000 7fb16a7ed700 1 -- start start 2023-10-06T01:42:07.900 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.898+0000 7fb16a7ed700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb1640afde0 0x7fb1641620e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:07.900 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.898+0000 7fb16a7ed700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1640cc980 con 0x7fb16415b1a0 2023-10-06T01:42:07.900 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.898+0000 7fb16a7ed700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1640b0330 con 0x7fb164155120 2023-10-06T01:42:07.900 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.898+0000 7fb16a7ed700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb16415be40 con 0x7fb1640afde0 2023-10-06T01:42:07.900 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.898+0000 7fb169fec700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fb1640afde0 0x7fb1641620e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:07.900 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.898+0000 7fb1697eb700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb16415b1a0 0x7fb164154a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52092/0 (socket says 172.21.15.27:52092) 2023-10-06T01:42:07.901 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.898+0000 7fb1697eb700 1 -- 172.21.15.27:0/3862203718 learned_addr learned my addr 172.21.15.27:0/3862203718 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:07.901 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.898+0000 7fb169fec700 1 -- 172.21.15.27:0/3862203718 >> v1:172.21.15.124:6789/0 conn(0x7fb164155120 legacy=0x7fb1640af6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:07.901 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.898+0000 7fb169fec700 1 -- 172.21.15.27:0/3862203718 >> v1:172.21.15.27:6789/0 conn(0x7fb16415b1a0 legacy=0x7fb164154a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:07.901 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.898+0000 7fb169fec700 1 -- 172.21.15.27:0/3862203718 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1640b0610 con 0x7fb1640afde0 2023-10-06T01:42:07.901 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.899+0000 7fb169fec700 1 --2- 172.21.15.27:0/3862203718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1640afde0 0x7fb1641620e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fb15c009e60 tx=0x7fb15c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2b0f5c2b4ec5634 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:07.901 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.899+0000 7fb15bfff700 1 -- 172.21.15.27:0/3862203718 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15c00e070 con 0x7fb1640afde0 2023-10-06T01:42:07.901 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.899+0000 7fb15bfff700 1 -- 172.21.15.27:0/3862203718 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb15c00bc30 con 0x7fb1640afde0 2023-10-06T01:42:07.901 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.899+0000 7fb15bfff700 1 -- 172.21.15.27:0/3862203718 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb15c012430 con 0x7fb1640afde0 2023-10-06T01:42:07.904 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.899+0000 7fb16a7ed700 1 -- 172.21.15.27:0/3862203718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1640afde0 msgr2=0x7fb1641620e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:07.904 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.899+0000 7fb16a7ed700 1 --2- 172.21.15.27:0/3862203718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1640afde0 0x7fb1641620e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fb15c009e60 tx=0x7fb15c009780 comp rx=0 tx=0).stop 2023-10-06T01:42:07.905 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.899+0000 7fb16a7ed700 1 -- 172.21.15.27:0/3862203718 shutdown_connections 2023-10-06T01:42:07.905 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.899+0000 7fb16a7ed700 1 --2- 172.21.15.27:0/3862203718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1640afde0 0x7fb1641620e0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:07.905 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.899+0000 7fb16a7ed700 1 -- 172.21.15.27:0/3862203718 >> 172.21.15.27:0/3862203718 conn(0x7fb1640ce320 msgr2=0x7fb1640ce720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:07.905 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.899+0000 7fb16a7ed700 1 -- 172.21.15.27:0/3862203718 shutdown_connections 2023-10-06T01:42:07.905 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.899+0000 7fb16a7ed700 1 -- 172.21.15.27:0/3862203718 wait complete. 2023-10-06T01:42:07.905 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.900+0000 7fb16a7ed700 1 Processor -- start 2023-10-06T01:42:07.905 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.900+0000 7fb16a7ed700 1 -- start start 2023-10-06T01:42:07.905 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.901+0000 7fb16a7ed700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb1640afde0 0x7fb164158010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:07.905 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.901+0000 7fb16a7ed700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb164155120 0x7fb164151f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:07.905 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.901+0000 7fb16a7ed700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb16415b1a0 0x7fb1640b2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:07.905 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.901+0000 7fb16a7ed700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb164167740 con 0x7fb164155120 2023-10-06T01:42:07.906 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.901+0000 7fb16a7ed700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb164167450 con 0x7fb16415b1a0 2023-10-06T01:42:07.906 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.901+0000 7fb16a7ed700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1640b2850 con 0x7fb1640afde0 2023-10-06T01:42:07.906 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.901+0000 7fb168fea700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb164155120 0x7fb164151f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:07.906 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.901+0000 7fb168fea700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb164155120 0x7fb164151f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37456/0 (socket says 172.21.15.27:37456) 2023-10-06T01:42:07.906 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.901+0000 7fb168fea700 1 -- 172.21.15.27:0/308745329 learned_addr learned my addr 172.21.15.27:0/308745329 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:07.906 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.902+0000 7fb169fec700 1 --2- 172.21.15.27:0/308745329 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb16415b1a0 0x7fb1640b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:07.906 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.902+0000 7fb1697eb700 1 --2- 172.21.15.27:0/308745329 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb1640afde0 0x7fb164158010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:07.906 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.902+0000 7fb168fea700 1 -- 172.21.15.27:0/308745329 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb1640afde0 msgr2=0x7fb164158010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:07.906 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.902+0000 7fb168fea700 1 --2- 172.21.15.27:0/308745329 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb1640afde0 0x7fb164158010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:07.906 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.902+0000 7fb168fea700 1 -- 172.21.15.27:0/308745329 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb16415b1a0 msgr2=0x7fb1640b2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:07.906 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.902+0000 7fb168fea700 1 --2- 172.21.15.27:0/308745329 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb16415b1a0 0x7fb1640b2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:07.906 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.902+0000 7fb168fea700 1 -- 172.21.15.27:0/308745329 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb15c009a20 con 0x7fb164155120 2023-10-06T01:42:07.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.902+0000 7fb169fec700 1 --2- 172.21.15.27:0/308745329 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb16415b1a0 0x7fb1640b2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:42:07.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.902+0000 7fb168fea700 1 --2- 172.21.15.27:0/308745329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb164155120 0x7fb164151f80 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb15400c590 tx=0x7fb154007f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:07.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.902+0000 7fb1597fa700 1 -- 172.21.15.27:0/308745329 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb154011590 con 0x7fb164155120 2023-10-06T01:42:07.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.902+0000 7fb1597fa700 1 -- 172.21.15.27:0/308745329 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1540116f0 con 0x7fb164155120 2023-10-06T01:42:07.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.902+0000 7fb16a7ed700 1 -- 172.21.15.27:0/308745329 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1640b2b30 con 0x7fb164155120 2023-10-06T01:42:07.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.902+0000 7fb1597fa700 1 -- 172.21.15.27:0/308745329 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb154011ab0 con 0x7fb164155120 2023-10-06T01:42:07.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.902+0000 7fb16a7ed700 1 -- 172.21.15.27:0/308745329 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1640b2560 con 0x7fb164155120 2023-10-06T01:42:07.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.903+0000 7fb16a7ed700 1 -- 172.21.15.27:0/308745329 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1640cf120 con 0x7fb164155120 2023-10-06T01:42:07.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.903+0000 7fb1597fa700 1 -- 172.21.15.27:0/308745329 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb15401c460 con 0x7fb164155120 2023-10-06T01:42:07.908 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.904+0000 7fb1597fa700 1 --2- 172.21.15.27:0/308745329 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb148066f30 0x7fb1480693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:07.908 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.904+0000 7fb1597fa700 1 -- 172.21.15.27:0/308745329 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fb1540844b0 con 0x7fb164155120 2023-10-06T01:42:07.908 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.904+0000 7fb1697eb700 1 --2- 172.21.15.27:0/308745329 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb148066f30 0x7fb1480693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:07.908 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.905+0000 7fb1697eb700 1 --2- 172.21.15.27:0/308745329 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb148066f30 0x7fb1480693e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb1600016e0 tx=0x7fb160009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:07.909 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:07.908+0000 7fb1597fa700 1 -- 172.21.15.27:0/308745329 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb1540515c0 con 0x7fb164155120 2023-10-06T01:42:08.096 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.095+0000 7fb16a7ed700 1 -- 172.21.15.27:0/308745329 --> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb16404c3c0 con 0x7fb148066f30 2023-10-06T01:42:08.104 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.103+0000 7fb1597fa700 1 -- 172.21.15.27:0/308745329 <== mgr.4100 v2:172.21.15.124:6832/90278 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290002 (secure 0 0 0) 0x7fb16404c3c0 con 0x7fb148066f30 2023-10-06T01:42:08.108 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:08.110 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.106+0000 7fb16a7ed700 1 -- 172.21.15.27:0/308745329 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb148066f30 msgr2=0x7fb1480693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:08.110 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.106+0000 7fb16a7ed700 1 --2- 172.21.15.27:0/308745329 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb148066f30 0x7fb1480693e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb1600016e0 tx=0x7fb160009040 comp rx=0 tx=0).stop 2023-10-06T01:42:08.110 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.106+0000 7fb16a7ed700 1 -- 172.21.15.27:0/308745329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb164155120 msgr2=0x7fb164151f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:08.110 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.106+0000 7fb16a7ed700 1 --2- 172.21.15.27:0/308745329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb164155120 0x7fb164151f80 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb15400c590 tx=0x7fb154007f30 comp rx=0 tx=0).stop 2023-10-06T01:42:08.110 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.107+0000 7fb16a7ed700 1 -- 172.21.15.27:0/308745329 shutdown_connections 2023-10-06T01:42:08.110 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.107+0000 7fb16a7ed700 1 --2- 172.21.15.27:0/308745329 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb148066f30 0x7fb1480693e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:08.111 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.107+0000 7fb16a7ed700 1 --2- 172.21.15.27:0/308745329 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb1640afde0 0x7fb164158010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:08.111 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.107+0000 7fb16a7ed700 1 --2- 172.21.15.27:0/308745329 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb16415b1a0 0x7fb1640b2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:08.111 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.107+0000 7fb16a7ed700 1 --2- 172.21.15.27:0/308745329 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb164155120 0x7fb164151f80 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:08.111 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.107+0000 7fb16a7ed700 1 -- 172.21.15.27:0/308745329 >> 172.21.15.27:0/308745329 conn(0x7fb1640ce320 msgr2=0x7fb164153200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:08.111 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.107+0000 7fb16a7ed700 1 -- 172.21.15.27:0/308745329 shutdown_connections 2023-10-06T01:42:08.111 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.107+0000 7fb16a7ed700 1 -- 172.21.15.27:0/308745329 wait complete. 2023-10-06T01:42:08.111 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-06T01:42:08.120 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":55,"stamp":"2023-10-06T01:42:08.051518+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592674,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2429,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":210,"acting":210,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":158,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2382184,"kb_used_data":3880,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752592536,"statfs":{"total":773094113280,"available":770654756864,"internally_reserved":0,"allocated":3973120,"data_stored":2282452,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001469"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":8,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.658560+0000","last_change":"2023-10-06T01:42:03.658560+0000","last_active":"2023-10-06T01:42:03.658560+0000","last_peered":"2023-10-06T01:42:03.658560+0000","last_clean":"2023-10-06T01:42:03.658560+0000","last_became_active":"2023-10-06T01:42:03.658288+0000","last_became_peered":"2023-10-06T01:42:03.658288+0000","last_unstale":"2023-10-06T01:42:03.658560+0000","last_undegraded":"2023-10-06T01:42:03.658560+0000","last_fullsized":"2023-10-06T01:42:03.658560+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:43:09.292191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":8,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.649498+0000","last_change":"2023-10-06T01:42:03.649498+0000","last_active":"2023-10-06T01:42:03.649498+0000","last_peered":"2023-10-06T01:42:03.649498+0000","last_clean":"2023-10-06T01:42:03.649498+0000","last_became_active":"2023-10-06T01:42:03.649252+0000","last_became_peered":"2023-10-06T01:42:03.649252+0000","last_unstale":"2023-10-06T01:42:03.649498+0000","last_undegraded":"2023-10-06T01:42:03.649498+0000","last_fullsized":"2023-10-06T01:42:03.649498+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:17:06.578187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.662639+0000","last_change":"2023-10-06T01:42:03.655756+0000","last_active":"2023-10-06T01:42:03.662639+0000","last_peered":"2023-10-06T01:42:03.662639+0000","last_clean":"2023-10-06T01:42:03.662639+0000","last_became_active":"2023-10-06T01:42:03.655266+0000","last_became_peered":"2023-10-06T01:42:03.655266+0000","last_unstale":"2023-10-06T01:42:03.662639+0000","last_undegraded":"2023-10-06T01:42:03.662639+0000","last_fullsized":"2023-10-06T01:42:03.662639+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:34:59.205863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.663707+0000","last_change":"2023-10-06T01:42:03.663219+0000","last_active":"2023-10-06T01:42:03.663707+0000","last_peered":"2023-10-06T01:42:03.663707+0000","last_clean":"2023-10-06T01:42:03.663707+0000","last_became_active":"2023-10-06T01:42:03.662554+0000","last_became_peered":"2023-10-06T01:42:03.662554+0000","last_unstale":"2023-10-06T01:42:03.663707+0000","last_undegraded":"2023-10-06T01:42:03.663707+0000","last_fullsized":"2023-10-06T01:42:03.663707+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:02:22.174614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.657184+0000","last_change":"2023-10-06T01:42:03.653519+0000","last_active":"2023-10-06T01:42:03.657184+0000","last_peered":"2023-10-06T01:42:03.657184+0000","last_clean":"2023-10-06T01:42:03.657184+0000","last_became_active":"2023-10-06T01:42:03.653239+0000","last_became_peered":"2023-10-06T01:42:03.653239+0000","last_unstale":"2023-10-06T01:42:03.657184+0000","last_undegraded":"2023-10-06T01:42:03.657184+0000","last_fullsized":"2023-10-06T01:42:03.657184+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:54:49.681004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":8,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.652127+0000","last_change":"2023-10-06T01:42:03.652127+0000","last_active":"2023-10-06T01:42:03.652127+0000","last_peered":"2023-10-06T01:42:03.652127+0000","last_clean":"2023-10-06T01:42:03.652127+0000","last_became_active":"2023-10-06T01:42:03.651874+0000","last_became_peered":"2023-10-06T01:42:03.651874+0000","last_unstale":"2023-10-06T01:42:03.652127+0000","last_undegraded":"2023-10-06T01:42:03.652127+0000","last_fullsized":"2023-10-06T01:42:03.652127+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:27:51.973115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.656719+0000","last_change":"2023-10-06T01:42:03.652972+0000","last_active":"2023-10-06T01:42:03.656719+0000","last_peered":"2023-10-06T01:42:03.656719+0000","last_clean":"2023-10-06T01:42:03.656719+0000","last_became_active":"2023-10-06T01:42:03.652724+0000","last_became_peered":"2023-10-06T01:42:03.652724+0000","last_unstale":"2023-10-06T01:42:03.656719+0000","last_undegraded":"2023-10-06T01:42:03.656719+0000","last_fullsized":"2023-10-06T01:42:03.656719+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:50:55.550267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":8,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.653994+0000","last_change":"2023-10-06T01:42:03.653994+0000","last_active":"2023-10-06T01:42:03.653994+0000","last_peered":"2023-10-06T01:42:03.653994+0000","last_clean":"2023-10-06T01:42:03.653994+0000","last_became_active":"2023-10-06T01:42:03.653698+0000","last_became_peered":"2023-10-06T01:42:03.653698+0000","last_unstale":"2023-10-06T01:42:03.653994+0000","last_undegraded":"2023-10-06T01:42:03.653994+0000","last_fullsized":"2023-10-06T01:42:03.653994+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:18:14.223068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.662622+0000","last_change":"2023-10-06T01:42:03.655576+0000","last_active":"2023-10-06T01:42:03.662622+0000","last_peered":"2023-10-06T01:42:03.662622+0000","last_clean":"2023-10-06T01:42:03.662622+0000","last_became_active":"2023-10-06T01:42:03.655228+0000","last_became_peered":"2023-10-06T01:42:03.655228+0000","last_unstale":"2023-10-06T01:42:03.662622+0000","last_undegraded":"2023-10-06T01:42:03.662622+0000","last_fullsized":"2023-10-06T01:42:03.662622+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:17:06.578187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.665099+0000","last_change":"2023-10-06T01:42:03.661575+0000","last_active":"2023-10-06T01:42:03.665099+0000","last_peered":"2023-10-06T01:42:03.665099+0000","last_clean":"2023-10-06T01:42:03.665099+0000","last_became_active":"2023-10-06T01:42:03.661295+0000","last_became_peered":"2023-10-06T01:42:03.661295+0000","last_unstale":"2023-10-06T01:42:03.665099+0000","last_undegraded":"2023-10-06T01:42:03.665099+0000","last_fullsized":"2023-10-06T01:42:03.665099+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:43:09.292191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.663675+0000","last_change":"2023-10-06T01:42:03.660758+0000","last_active":"2023-10-06T01:42:03.663675+0000","last_peered":"2023-10-06T01:42:03.663675+0000","last_clean":"2023-10-06T01:42:03.663675+0000","last_became_active":"2023-10-06T01:42:03.660504+0000","last_became_peered":"2023-10-06T01:42:03.660504+0000","last_unstale":"2023-10-06T01:42:03.663675+0000","last_undegraded":"2023-10-06T01:42:03.663675+0000","last_fullsized":"2023-10-06T01:42:03.663675+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:55:56.347224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.662381+0000","last_change":"2023-10-06T01:42:03.648412+0000","last_active":"2023-10-06T01:42:03.662381+0000","last_peered":"2023-10-06T01:42:03.662381+0000","last_clean":"2023-10-06T01:42:03.662381+0000","last_became_active":"2023-10-06T01:42:03.648159+0000","last_became_peered":"2023-10-06T01:42:03.648159+0000","last_unstale":"2023-10-06T01:42:03.662381+0000","last_undegraded":"2023-10-06T01:42:03.662381+0000","last_fullsized":"2023-10-06T01:42:03.662381+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:47:42.520312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.663575+0000","last_change":"2023-10-06T01:42:03.659895+0000","last_active":"2023-10-06T01:42:03.663575+0000","last_peered":"2023-10-06T01:42:03.663575+0000","last_clean":"2023-10-06T01:42:03.663575+0000","last_became_active":"2023-10-06T01:42:03.659637+0000","last_became_peered":"2023-10-06T01:42:03.659637+0000","last_unstale":"2023-10-06T01:42:03.663575+0000","last_undegraded":"2023-10-06T01:42:03.663575+0000","last_fullsized":"2023-10-06T01:42:03.663575+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:17:15.841349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.665333+0000","last_change":"2023-10-06T01:42:03.661480+0000","last_active":"2023-10-06T01:42:03.665333+0000","last_peered":"2023-10-06T01:42:03.665333+0000","last_clean":"2023-10-06T01:42:03.665333+0000","last_became_active":"2023-10-06T01:42:03.660853+0000","last_became_peered":"2023-10-06T01:42:03.660853+0000","last_unstale":"2023-10-06T01:42:03.665333+0000","last_undegraded":"2023-10-06T01:42:03.665333+0000","last_fullsized":"2023-10-06T01:42:03.665333+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:16:29.730704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.652575+0000","last_change":"2023-10-06T01:42:03.650963+0000","last_active":"2023-10-06T01:42:03.652575+0000","last_peered":"2023-10-06T01:42:03.652575+0000","last_clean":"2023-10-06T01:42:03.652575+0000","last_became_active":"2023-10-06T01:42:03.650699+0000","last_became_peered":"2023-10-06T01:42:03.650699+0000","last_unstale":"2023-10-06T01:42:03.652575+0000","last_undegraded":"2023-10-06T01:42:03.652575+0000","last_fullsized":"2023-10-06T01:42:03.652575+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:56:10.182021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.660796+0000","last_change":"2023-10-06T01:42:03.657365+0000","last_active":"2023-10-06T01:42:03.660796+0000","last_peered":"2023-10-06T01:42:03.660796+0000","last_clean":"2023-10-06T01:42:03.660796+0000","last_became_active":"2023-10-06T01:42:03.654780+0000","last_became_peered":"2023-10-06T01:42:03.654780+0000","last_unstale":"2023-10-06T01:42:03.660796+0000","last_undegraded":"2023-10-06T01:42:03.660796+0000","last_fullsized":"2023-10-06T01:42:03.660796+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:01:14.549337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.659265+0000","last_change":"2023-10-06T01:42:03.654812+0000","last_active":"2023-10-06T01:42:03.659265+0000","last_peered":"2023-10-06T01:42:03.659265+0000","last_clean":"2023-10-06T01:42:03.659265+0000","last_became_active":"2023-10-06T01:42:03.654536+0000","last_became_peered":"2023-10-06T01:42:03.654536+0000","last_unstale":"2023-10-06T01:42:03.659265+0000","last_undegraded":"2023-10-06T01:42:03.659265+0000","last_fullsized":"2023-10-06T01:42:03.659265+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:50:55.550267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.666452+0000","last_change":"2023-10-06T01:42:03.663358+0000","last_active":"2023-10-06T01:42:03.666452+0000","last_peered":"2023-10-06T01:42:03.666452+0000","last_clean":"2023-10-06T01:42:03.666452+0000","last_became_active":"2023-10-06T01:42:03.663083+0000","last_became_peered":"2023-10-06T01:42:03.663083+0000","last_unstale":"2023-10-06T01:42:03.666452+0000","last_undegraded":"2023-10-06T01:42:03.666452+0000","last_fullsized":"2023-10-06T01:42:03.666452+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:10:27.590638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.651492+0000","last_change":"2023-10-06T01:42:03.649322+0000","last_active":"2023-10-06T01:42:03.651492+0000","last_peered":"2023-10-06T01:42:03.651492+0000","last_clean":"2023-10-06T01:42:03.651492+0000","last_became_active":"2023-10-06T01:42:03.649033+0000","last_became_peered":"2023-10-06T01:42:03.649033+0000","last_unstale":"2023-10-06T01:42:03.651492+0000","last_undegraded":"2023-10-06T01:42:03.651492+0000","last_fullsized":"2023-10-06T01:42:03.651492+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:48:48.815450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.664669+0000","last_change":"2023-10-06T01:42:03.662781+0000","last_active":"2023-10-06T01:42:03.664669+0000","last_peered":"2023-10-06T01:42:03.664669+0000","last_clean":"2023-10-06T01:42:03.664669+0000","last_became_active":"2023-10-06T01:42:03.662508+0000","last_became_peered":"2023-10-06T01:42:03.662508+0000","last_unstale":"2023-10-06T01:42:03.664669+0000","last_undegraded":"2023-10-06T01:42:03.664669+0000","last_fullsized":"2023-10-06T01:42:03.664669+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:03:49.166875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.656147+0000","last_change":"2023-10-06T01:42:03.654075+0000","last_active":"2023-10-06T01:42:03.656147+0000","last_peered":"2023-10-06T01:42:03.656147+0000","last_clean":"2023-10-06T01:42:03.656147+0000","last_became_active":"2023-10-06T01:42:03.653714+0000","last_became_peered":"2023-10-06T01:42:03.653714+0000","last_unstale":"2023-10-06T01:42:03.656147+0000","last_undegraded":"2023-10-06T01:42:03.656147+0000","last_fullsized":"2023-10-06T01:42:03.656147+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:17:06.578187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.665387+0000","last_change":"2023-10-06T01:42:03.661953+0000","last_active":"2023-10-06T01:42:03.665387+0000","last_peered":"2023-10-06T01:42:03.665387+0000","last_clean":"2023-10-06T01:42:03.665387+0000","last_became_active":"2023-10-06T01:42:03.661648+0000","last_became_peered":"2023-10-06T01:42:03.661648+0000","last_unstale":"2023-10-06T01:42:03.665387+0000","last_undegraded":"2023-10-06T01:42:03.665387+0000","last_fullsized":"2023-10-06T01:42:03.665387+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:44:54.695371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.659193+0000","last_change":"2023-10-06T01:42:03.653326+0000","last_active":"2023-10-06T01:42:03.659193+0000","last_peered":"2023-10-06T01:42:03.659193+0000","last_clean":"2023-10-06T01:42:03.659193+0000","last_became_active":"2023-10-06T01:42:03.652745+0000","last_became_peered":"2023-10-06T01:42:03.652745+0000","last_unstale":"2023-10-06T01:42:03.659193+0000","last_undegraded":"2023-10-06T01:42:03.659193+0000","last_fullsized":"2023-10-06T01:42:03.659193+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:14:40.847505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.659789+0000","last_change":"2023-10-06T01:42:03.658697+0000","last_active":"2023-10-06T01:42:03.659789+0000","last_peered":"2023-10-06T01:42:03.659789+0000","last_clean":"2023-10-06T01:42:03.659789+0000","last_became_active":"2023-10-06T01:42:03.658387+0000","last_became_peered":"2023-10-06T01:42:03.658387+0000","last_unstale":"2023-10-06T01:42:03.659789+0000","last_undegraded":"2023-10-06T01:42:03.659789+0000","last_fullsized":"2023-10-06T01:42:03.659789+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:20:56.966643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.660970+0000","last_change":"2023-10-06T01:42:03.658742+0000","last_active":"2023-10-06T01:42:03.660970+0000","last_peered":"2023-10-06T01:42:03.660970+0000","last_clean":"2023-10-06T01:42:03.660970+0000","last_became_active":"2023-10-06T01:42:03.658367+0000","last_became_peered":"2023-10-06T01:42:03.658367+0000","last_unstale":"2023-10-06T01:42:03.660970+0000","last_undegraded":"2023-10-06T01:42:03.660970+0000","last_fullsized":"2023-10-06T01:42:03.660970+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:43:37.340057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":8,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.655189+0000","last_change":"2023-10-06T01:42:03.655189+0000","last_active":"2023-10-06T01:42:03.655189+0000","last_peered":"2023-10-06T01:42:03.655189+0000","last_clean":"2023-10-06T01:42:03.655189+0000","last_became_active":"2023-10-06T01:42:03.654961+0000","last_became_peered":"2023-10-06T01:42:03.654961+0000","last_unstale":"2023-10-06T01:42:03.655189+0000","last_undegraded":"2023-10-06T01:42:03.655189+0000","last_fullsized":"2023-10-06T01:42:03.655189+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:56:53.849323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.654459+0000","last_change":"2023-10-06T01:42:03.648052+0000","last_active":"2023-10-06T01:42:03.654459+0000","last_peered":"2023-10-06T01:42:03.654459+0000","last_clean":"2023-10-06T01:42:03.654459+0000","last_became_active":"2023-10-06T01:42:03.647779+0000","last_became_peered":"2023-10-06T01:42:03.647779+0000","last_unstale":"2023-10-06T01:42:03.654459+0000","last_undegraded":"2023-10-06T01:42:03.654459+0000","last_fullsized":"2023-10-06T01:42:03.654459+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:48:56.904883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.661672+0000","last_change":"2023-10-06T01:42:03.658055+0000","last_active":"2023-10-06T01:42:03.661672+0000","last_peered":"2023-10-06T01:42:03.661672+0000","last_clean":"2023-10-06T01:42:03.661672+0000","last_became_active":"2023-10-06T01:42:03.657677+0000","last_became_peered":"2023-10-06T01:42:03.657677+0000","last_unstale":"2023-10-06T01:42:03.661672+0000","last_undegraded":"2023-10-06T01:42:03.661672+0000","last_fullsized":"2023-10-06T01:42:03.661672+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:14:40.847505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.653088+0000","last_change":"2023-10-06T01:42:03.647016+0000","last_active":"2023-10-06T01:42:03.653088+0000","last_peered":"2023-10-06T01:42:03.653088+0000","last_clean":"2023-10-06T01:42:03.653088+0000","last_became_active":"2023-10-06T01:42:03.646725+0000","last_became_peered":"2023-10-06T01:42:03.646725+0000","last_unstale":"2023-10-06T01:42:03.653088+0000","last_undegraded":"2023-10-06T01:42:03.653088+0000","last_fullsized":"2023-10-06T01:42:03.653088+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:04:21.094848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.655313+0000","last_change":"2023-10-06T01:42:03.651596+0000","last_active":"2023-10-06T01:42:03.655313+0000","last_peered":"2023-10-06T01:42:03.655313+0000","last_clean":"2023-10-06T01:42:03.655313+0000","last_became_active":"2023-10-06T01:42:03.651157+0000","last_became_peered":"2023-10-06T01:42:03.651157+0000","last_unstale":"2023-10-06T01:42:03.655313+0000","last_undegraded":"2023-10-06T01:42:03.655313+0000","last_fullsized":"2023-10-06T01:42:03.655313+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:20:16.734800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.659214+0000","last_change":"2023-10-06T01:42:03.653580+0000","last_active":"2023-10-06T01:42:03.659214+0000","last_peered":"2023-10-06T01:42:03.659214+0000","last_clean":"2023-10-06T01:42:03.659214+0000","last_became_active":"2023-10-06T01:42:03.653075+0000","last_became_peered":"2023-10-06T01:42:03.653075+0000","last_unstale":"2023-10-06T01:42:03.659214+0000","last_undegraded":"2023-10-06T01:42:03.659214+0000","last_fullsized":"2023-10-06T01:42:03.659214+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:37:15.263123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":8,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.647572+0000","last_change":"2023-10-06T01:42:03.647572+0000","last_active":"2023-10-06T01:42:03.647572+0000","last_peered":"2023-10-06T01:42:03.647572+0000","last_clean":"2023-10-06T01:42:03.647572+0000","last_became_active":"2023-10-06T01:42:03.647288+0000","last_became_peered":"2023-10-06T01:42:03.647288+0000","last_unstale":"2023-10-06T01:42:03.647572+0000","last_undegraded":"2023-10-06T01:42:03.647572+0000","last_fullsized":"2023-10-06T01:42:03.647572+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:06:42.668497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.660992+0000","last_change":"2023-10-06T01:42:03.656983+0000","last_active":"2023-10-06T01:42:03.660992+0000","last_peered":"2023-10-06T01:42:03.660992+0000","last_clean":"2023-10-06T01:42:03.660992+0000","last_became_active":"2023-10-06T01:42:03.654594+0000","last_became_peered":"2023-10-06T01:42:03.654594+0000","last_unstale":"2023-10-06T01:42:03.660992+0000","last_undegraded":"2023-10-06T01:42:03.660992+0000","last_fullsized":"2023-10-06T01:42:03.660992+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:47:42.520312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"30'4","reported_seq":14,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.664143+0000","last_change":"2023-10-06T01:42:02.650214+0000","last_active":"2023-10-06T01:42:03.664143+0000","last_peered":"2023-10-06T01:42:03.664143+0000","last_clean":"2023-10-06T01:42:03.664143+0000","last_became_active":"2023-10-06T01:42:02.649908+0000","last_became_peered":"2023-10-06T01:42:02.649908+0000","last_unstale":"2023-10-06T01:42:03.664143+0000","last_undegraded":"2023-10-06T01:42:03.664143+0000","last_fullsized":"2023-10-06T01:42:03.664143+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:26:05.275378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":554,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":8,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.654697+0000","last_change":"2023-10-06T01:42:03.654697+0000","last_active":"2023-10-06T01:42:03.654697+0000","last_peered":"2023-10-06T01:42:03.654697+0000","last_clean":"2023-10-06T01:42:03.654697+0000","last_became_active":"2023-10-06T01:42:03.654384+0000","last_became_peered":"2023-10-06T01:42:03.654384+0000","last_unstale":"2023-10-06T01:42:03.654697+0000","last_undegraded":"2023-10-06T01:42:03.654697+0000","last_fullsized":"2023-10-06T01:42:03.654697+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:10:24.478901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"30'1","reported_seq":11,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.655960+0000","last_change":"2023-10-06T01:42:02.646451+0000","last_active":"2023-10-06T01:42:03.655960+0000","last_peered":"2023-10-06T01:42:03.655960+0000","last_clean":"2023-10-06T01:42:03.655960+0000","last_became_active":"2023-10-06T01:42:02.646182+0000","last_became_peered":"2023-10-06T01:42:02.646182+0000","last_unstale":"2023-10-06T01:42:03.655960+0000","last_undegraded":"2023-10-06T01:42:03.655960+0000","last_fullsized":"2023-10-06T01:42:03.655960+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:20:15.728651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.665477+0000","last_change":"2023-10-06T01:42:03.663136+0000","last_active":"2023-10-06T01:42:03.665477+0000","last_peered":"2023-10-06T01:42:03.665477+0000","last_clean":"2023-10-06T01:42:03.665477+0000","last_became_active":"2023-10-06T01:42:03.662468+0000","last_became_peered":"2023-10-06T01:42:03.662468+0000","last_unstale":"2023-10-06T01:42:03.665477+0000","last_undegraded":"2023-10-06T01:42:03.665477+0000","last_fullsized":"2023-10-06T01:42:03.665477+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:58:29.344469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"30'2","reported_seq":11,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.660139+0000","last_change":"2023-10-06T01:42:02.649210+0000","last_active":"2023-10-06T01:42:03.660139+0000","last_peered":"2023-10-06T01:42:03.660139+0000","last_clean":"2023-10-06T01:42:03.660139+0000","last_became_active":"2023-10-06T01:42:02.648923+0000","last_became_peered":"2023-10-06T01:42:02.648923+0000","last_unstale":"2023-10-06T01:42:03.660139+0000","last_undegraded":"2023-10-06T01:42:03.660139+0000","last_fullsized":"2023-10-06T01:42:03.660139+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:02:21.168465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":8,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.653802+0000","last_change":"2023-10-06T01:42:03.653802+0000","last_active":"2023-10-06T01:42:03.653802+0000","last_peered":"2023-10-06T01:42:03.653802+0000","last_clean":"2023-10-06T01:42:03.653802+0000","last_became_active":"2023-10-06T01:42:03.653504+0000","last_became_peered":"2023-10-06T01:42:03.653504+0000","last_unstale":"2023-10-06T01:42:03.653802+0000","last_undegraded":"2023-10-06T01:42:03.653802+0000","last_fullsized":"2023-10-06T01:42:03.653802+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:56:10.182021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"30'2","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.662121+0000","last_change":"2023-10-06T01:42:02.644361+0000","last_active":"2023-10-06T01:42:03.662121+0000","last_peered":"2023-10-06T01:42:03.662121+0000","last_clean":"2023-10-06T01:42:03.662121+0000","last_became_active":"2023-10-06T01:42:02.644142+0000","last_became_peered":"2023-10-06T01:42:02.644142+0000","last_unstale":"2023-10-06T01:42:03.662121+0000","last_undegraded":"2023-10-06T01:42:03.662121+0000","last_fullsized":"2023-10-06T01:42:03.662121+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:47:41.514163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.646780+0000","last_change":"2023-10-06T01:42:03.641991+0000","last_active":"2023-10-06T01:42:03.646780+0000","last_peered":"2023-10-06T01:42:03.646780+0000","last_clean":"2023-10-06T01:42:03.646780+0000","last_became_active":"2023-10-06T01:42:03.641759+0000","last_became_peered":"2023-10-06T01:42:03.641759+0000","last_unstale":"2023-10-06T01:42:03.646780+0000","last_undegraded":"2023-10-06T01:42:03.646780+0000","last_fullsized":"2023-10-06T01:42:03.646780+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:06:21.581530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"30'1","reported_seq":11,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.663011+0000","last_change":"2023-10-06T01:42:02.653079+0000","last_active":"2023-10-06T01:42:03.663011+0000","last_peered":"2023-10-06T01:42:03.663011+0000","last_clean":"2023-10-06T01:42:03.663011+0000","last_became_active":"2023-10-06T01:42:02.652801+0000","last_became_peered":"2023-10-06T01:42:02.652801+0000","last_unstale":"2023-10-06T01:42:03.663011+0000","last_undegraded":"2023-10-06T01:42:03.663011+0000","last_fullsized":"2023-10-06T01:42:03.663011+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:48:55.898734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.656734+0000","last_change":"2023-10-06T01:42:03.653900+0000","last_active":"2023-10-06T01:42:03.656734+0000","last_peered":"2023-10-06T01:42:03.656734+0000","last_clean":"2023-10-06T01:42:03.656734+0000","last_became_active":"2023-10-06T01:42:03.653610+0000","last_became_peered":"2023-10-06T01:42:03.653610+0000","last_unstale":"2023-10-06T01:42:03.656734+0000","last_undegraded":"2023-10-06T01:42:03.656734+0000","last_fullsized":"2023-10-06T01:42:03.656734+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:23:52.767666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"30'1","reported_seq":11,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.661999+0000","last_change":"2023-10-06T01:42:02.650466+0000","last_active":"2023-10-06T01:42:03.661999+0000","last_peered":"2023-10-06T01:42:03.661999+0000","last_clean":"2023-10-06T01:42:03.661999+0000","last_became_active":"2023-10-06T01:42:02.650215+0000","last_became_peered":"2023-10-06T01:42:02.650215+0000","last_unstale":"2023-10-06T01:42:03.661999+0000","last_undegraded":"2023-10-06T01:42:03.661999+0000","last_fullsized":"2023-10-06T01:42:03.661999+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:02:21.168465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.665299+0000","last_change":"2023-10-06T01:42:03.661100+0000","last_active":"2023-10-06T01:42:03.665299+0000","last_peered":"2023-10-06T01:42:03.665299+0000","last_clean":"2023-10-06T01:42:03.665299+0000","last_became_active":"2023-10-06T01:42:03.652425+0000","last_became_peered":"2023-10-06T01:42:03.652425+0000","last_unstale":"2023-10-06T01:42:03.665299+0000","last_undegraded":"2023-10-06T01:42:03.665299+0000","last_fullsized":"2023-10-06T01:42:03.665299+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:12:50.214916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-06T01:42:02.654661+0000","last_change":"2023-10-06T01:42:02.651745+0000","last_active":"2023-10-06T01:42:02.654661+0000","last_peered":"2023-10-06T01:42:02.654661+0000","last_clean":"2023-10-06T01:42:02.654661+0000","last_became_active":"2023-10-06T01:42:02.651455+0000","last_became_peered":"2023-10-06T01:42:02.651455+0000","last_unstale":"2023-10-06T01:42:02.654661+0000","last_undegraded":"2023-10-06T01:42:02.654661+0000","last_fullsized":"2023-10-06T01:42:02.654661+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:36:37.018597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":8,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.656442+0000","last_change":"2023-10-06T01:42:03.656442+0000","last_active":"2023-10-06T01:42:03.656442+0000","last_peered":"2023-10-06T01:42:03.656442+0000","last_clean":"2023-10-06T01:42:03.656442+0000","last_became_active":"2023-10-06T01:42:03.656218+0000","last_became_peered":"2023-10-06T01:42:03.656218+0000","last_unstale":"2023-10-06T01:42:03.656442+0000","last_undegraded":"2023-10-06T01:42:03.656442+0000","last_fullsized":"2023-10-06T01:42:03.656442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:16:29.730704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.644887+0000","last_change":"2023-10-06T01:42:02.645054+0000","last_active":"2023-10-06T01:42:03.644887+0000","last_peered":"2023-10-06T01:42:03.644887+0000","last_clean":"2023-10-06T01:42:03.644887+0000","last_became_active":"2023-10-06T01:42:02.644794+0000","last_became_peered":"2023-10-06T01:42:02.644794+0000","last_unstale":"2023-10-06T01:42:03.644887+0000","last_undegraded":"2023-10-06T01:42:03.644887+0000","last_fullsized":"2023-10-06T01:42:03.644887+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:14:39.841356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.663689+0000","last_change":"2023-10-06T01:42:03.662873+0000","last_active":"2023-10-06T01:42:03.663689+0000","last_peered":"2023-10-06T01:42:03.663689+0000","last_clean":"2023-10-06T01:42:03.663689+0000","last_became_active":"2023-10-06T01:42:03.662516+0000","last_became_peered":"2023-10-06T01:42:03.662516+0000","last_unstale":"2023-10-06T01:42:03.663689+0000","last_undegraded":"2023-10-06T01:42:03.663689+0000","last_fullsized":"2023-10-06T01:42:03.663689+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:15:04.488648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-06T01:42:02.650378+0000","last_change":"2023-10-06T01:42:02.647608+0000","last_active":"2023-10-06T01:42:02.650378+0000","last_peered":"2023-10-06T01:42:02.650378+0000","last_clean":"2023-10-06T01:42:02.650378+0000","last_became_active":"2023-10-06T01:42:02.647350+0000","last_became_peered":"2023-10-06T01:42:02.647350+0000","last_unstale":"2023-10-06T01:42:02.650378+0000","last_undegraded":"2023-10-06T01:42:02.650378+0000","last_fullsized":"2023-10-06T01:42:02.650378+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:27:50.966966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.660963+0000","last_change":"2023-10-06T01:42:03.657619+0000","last_active":"2023-10-06T01:42:03.660963+0000","last_peered":"2023-10-06T01:42:03.660963+0000","last_clean":"2023-10-06T01:42:03.660963+0000","last_became_active":"2023-10-06T01:42:03.657367+0000","last_became_peered":"2023-10-06T01:42:03.657367+0000","last_unstale":"2023-10-06T01:42:03.660963+0000","last_undegraded":"2023-10-06T01:42:03.660963+0000","last_fullsized":"2023-10-06T01:42:03.660963+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:34:59.205863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.646652+0000","last_change":"2023-10-06T01:42:02.644741+0000","last_active":"2023-10-06T01:42:03.646652+0000","last_peered":"2023-10-06T01:42:03.646652+0000","last_clean":"2023-10-06T01:42:03.646652+0000","last_became_active":"2023-10-06T01:42:02.644493+0000","last_became_peered":"2023-10-06T01:42:02.644493+0000","last_unstale":"2023-10-06T01:42:03.646652+0000","last_undegraded":"2023-10-06T01:42:03.646652+0000","last_fullsized":"2023-10-06T01:42:03.646652+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:04:20.088699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.662932+0000","last_change":"2023-10-06T01:42:03.658539+0000","last_active":"2023-10-06T01:42:03.662932+0000","last_peered":"2023-10-06T01:42:03.662932+0000","last_clean":"2023-10-06T01:42:03.662932+0000","last_became_active":"2023-10-06T01:42:03.658265+0000","last_became_peered":"2023-10-06T01:42:03.658265+0000","last_unstale":"2023-10-06T01:42:03.662932+0000","last_undegraded":"2023-10-06T01:42:03.662932+0000","last_fullsized":"2023-10-06T01:42:03.662932+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:01:14.549337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"30'3","reported_seq":13,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.654249+0000","last_change":"2023-10-06T01:42:02.645426+0000","last_active":"2023-10-06T01:42:03.654249+0000","last_peered":"2023-10-06T01:42:03.654249+0000","last_clean":"2023-10-06T01:42:03.654249+0000","last_became_active":"2023-10-06T01:42:02.645114+0000","last_became_peered":"2023-10-06T01:42:02.645114+0000","last_unstale":"2023-10-06T01:42:03.654249+0000","last_undegraded":"2023-10-06T01:42:03.654249+0000","last_fullsized":"2023-10-06T01:42:03.654249+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:03:48.160726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.663793+0000","last_change":"2023-10-06T01:42:03.651841+0000","last_active":"2023-10-06T01:42:03.663793+0000","last_peered":"2023-10-06T01:42:03.663793+0000","last_clean":"2023-10-06T01:42:03.663793+0000","last_became_active":"2023-10-06T01:42:03.651644+0000","last_became_peered":"2023-10-06T01:42:03.651644+0000","last_unstale":"2023-10-06T01:42:03.663793+0000","last_undegraded":"2023-10-06T01:42:03.663793+0000","last_fullsized":"2023-10-06T01:42:03.663793+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:50:31.746019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"30'6","reported_seq":15,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.656461+0000","last_change":"2023-10-06T01:42:02.649160+0000","last_active":"2023-10-06T01:42:03.656461+0000","last_peered":"2023-10-06T01:42:03.656461+0000","last_clean":"2023-10-06T01:42:03.656461+0000","last_became_active":"2023-10-06T01:42:02.648829+0000","last_became_peered":"2023-10-06T01:42:02.648829+0000","last_unstale":"2023-10-06T01:42:03.656461+0000","last_undegraded":"2023-10-06T01:42:03.656461+0000","last_fullsized":"2023-10-06T01:42:03.656461+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:06:41.662348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1536,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.662662+0000","last_change":"2023-10-06T01:42:03.655894+0000","last_active":"2023-10-06T01:42:03.662662+0000","last_peered":"2023-10-06T01:42:03.662662+0000","last_clean":"2023-10-06T01:42:03.662662+0000","last_became_active":"2023-10-06T01:42:03.655283+0000","last_became_peered":"2023-10-06T01:42:03.655283+0000","last_unstale":"2023-10-06T01:42:03.662662+0000","last_undegraded":"2023-10-06T01:42:03.662662+0000","last_fullsized":"2023-10-06T01:42:03.662662+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:37:15.263123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"30'2","reported_seq":11,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.658107+0000","last_change":"2023-10-06T01:42:02.642881+0000","last_active":"2023-10-06T01:42:03.658107+0000","last_peered":"2023-10-06T01:42:03.658107+0000","last_clean":"2023-10-06T01:42:03.658107+0000","last_became_active":"2023-10-06T01:42:02.642598+0000","last_became_peered":"2023-10-06T01:42:02.642598+0000","last_unstale":"2023-10-06T01:42:03.658107+0000","last_undegraded":"2023-10-06T01:42:03.658107+0000","last_fullsized":"2023-10-06T01:42:03.658107+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:03:48.160726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":8,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.649487+0000","last_change":"2023-10-06T01:42:03.649487+0000","last_active":"2023-10-06T01:42:03.649487+0000","last_peered":"2023-10-06T01:42:03.649487+0000","last_clean":"2023-10-06T01:42:03.649487+0000","last_became_active":"2023-10-06T01:42:03.649193+0000","last_became_peered":"2023-10-06T01:42:03.649193+0000","last_unstale":"2023-10-06T01:42:03.649487+0000","last_undegraded":"2023-10-06T01:42:03.649487+0000","last_fullsized":"2023-10-06T01:42:03.649487+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:17:15.841349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-06T01:42:02.654678+0000","last_change":"2023-10-06T01:42:02.652300+0000","last_active":"2023-10-06T01:42:02.654678+0000","last_peered":"2023-10-06T01:42:02.654678+0000","last_clean":"2023-10-06T01:42:02.654678+0000","last_became_active":"2023-10-06T01:42:02.652043+0000","last_became_peered":"2023-10-06T01:42:02.652043+0000","last_unstale":"2023-10-06T01:42:02.654678+0000","last_undegraded":"2023-10-06T01:42:02.654678+0000","last_fullsized":"2023-10-06T01:42:02.654678+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:43:36.333908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.659100+0000","last_change":"2023-10-06T01:42:03.649986+0000","last_active":"2023-10-06T01:42:03.659100+0000","last_peered":"2023-10-06T01:42:03.659100+0000","last_clean":"2023-10-06T01:42:03.659100+0000","last_became_active":"2023-10-06T01:42:03.649697+0000","last_became_peered":"2023-10-06T01:42:03.649697+0000","last_unstale":"2023-10-06T01:42:03.659100+0000","last_undegraded":"2023-10-06T01:42:03.659100+0000","last_fullsized":"2023-10-06T01:42:03.659100+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:07:16.583330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.637264+0000","last_change":"2023-10-06T01:42:02.650296+0000","last_active":"2023-10-06T01:42:03.637264+0000","last_peered":"2023-10-06T01:42:03.637264+0000","last_clean":"2023-10-06T01:42:03.637264+0000","last_became_active":"2023-10-06T01:42:02.649989+0000","last_became_peered":"2023-10-06T01:42:02.649989+0000","last_unstale":"2023-10-06T01:42:03.637264+0000","last_undegraded":"2023-10-06T01:42:03.637264+0000","last_fullsized":"2023-10-06T01:42:03.637264+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T10:52:46.616460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":8,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.658901+0000","last_change":"2023-10-06T01:42:03.658901+0000","last_active":"2023-10-06T01:42:03.658901+0000","last_peered":"2023-10-06T01:42:03.658901+0000","last_clean":"2023-10-06T01:42:03.658901+0000","last_became_active":"2023-10-06T01:42:03.658334+0000","last_became_peered":"2023-10-06T01:42:03.658334+0000","last_unstale":"2023-10-06T01:42:03.658901+0000","last_undegraded":"2023-10-06T01:42:03.658901+0000","last_fullsized":"2023-10-06T01:42:03.658901+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:23:52.767666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"30'8","reported_seq":18,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.664211+0000","last_change":"2023-10-06T01:42:02.649139+0000","last_active":"2023-10-06T01:42:03.664211+0000","last_peered":"2023-10-06T01:42:03.664211+0000","last_clean":"2023-10-06T01:42:03.664211+0000","last_became_active":"2023-10-06T01:42:02.648909+0000","last_became_peered":"2023-10-06T01:42:02.648909+0000","last_unstale":"2023-10-06T01:42:03.664211+0000","last_undegraded":"2023-10-06T01:42:03.664211+0000","last_fullsized":"2023-10-06T01:42:03.664211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:15:03.482499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":8,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.654313+0000","last_change":"2023-10-06T01:42:03.654313+0000","last_active":"2023-10-06T01:42:03.654313+0000","last_peered":"2023-10-06T01:42:03.654313+0000","last_clean":"2023-10-06T01:42:03.654313+0000","last_became_active":"2023-10-06T01:42:03.654030+0000","last_became_peered":"2023-10-06T01:42:03.654030+0000","last_unstale":"2023-10-06T01:42:03.654313+0000","last_undegraded":"2023-10-06T01:42:03.654313+0000","last_fullsized":"2023-10-06T01:42:03.654313+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:55:56.347224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"30'1","reported_seq":11,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.664091+0000","last_change":"2023-10-06T01:42:02.651772+0000","last_active":"2023-10-06T01:42:03.664091+0000","last_peered":"2023-10-06T01:42:03.664091+0000","last_clean":"2023-10-06T01:42:03.664091+0000","last_became_active":"2023-10-06T01:42:02.651497+0000","last_became_peered":"2023-10-06T01:42:02.651497+0000","last_unstale":"2023-10-06T01:42:03.664091+0000","last_undegraded":"2023-10-06T01:42:03.664091+0000","last_fullsized":"2023-10-06T01:42:03.664091+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:20:15.728651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.656892+0000","last_change":"2023-10-06T01:42:03.654398+0000","last_active":"2023-10-06T01:42:03.656892+0000","last_peered":"2023-10-06T01:42:03.656892+0000","last_clean":"2023-10-06T01:42:03.656892+0000","last_became_active":"2023-10-06T01:42:03.653718+0000","last_became_peered":"2023-10-06T01:42:03.653718+0000","last_unstale":"2023-10-06T01:42:03.656892+0000","last_undegraded":"2023-10-06T01:42:03.656892+0000","last_fullsized":"2023-10-06T01:42:03.656892+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:16:29.730704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"30'2","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.665258+0000","last_change":"2023-10-06T01:42:02.649849+0000","last_active":"2023-10-06T01:42:03.665258+0000","last_peered":"2023-10-06T01:42:03.665258+0000","last_clean":"2023-10-06T01:42:03.665258+0000","last_became_active":"2023-10-06T01:42:02.649590+0000","last_became_peered":"2023-10-06T01:42:02.649590+0000","last_unstale":"2023-10-06T01:42:03.665258+0000","last_undegraded":"2023-10-06T01:42:03.665258+0000","last_fullsized":"2023-10-06T01:42:03.665258+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:01:13.543188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.665428+0000","last_change":"2023-10-06T01:42:03.662501+0000","last_active":"2023-10-06T01:42:03.665428+0000","last_peered":"2023-10-06T01:42:03.665428+0000","last_clean":"2023-10-06T01:42:03.665428+0000","last_became_active":"2023-10-06T01:42:03.652350+0000","last_became_peered":"2023-10-06T01:42:03.652350+0000","last_unstale":"2023-10-06T01:42:03.665428+0000","last_undegraded":"2023-10-06T01:42:03.665428+0000","last_fullsized":"2023-10-06T01:42:03.665428+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:56:54.630363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"30'1","reported_seq":11,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.656960+0000","last_change":"2023-10-06T01:42:02.649844+0000","last_active":"2023-10-06T01:42:03.656960+0000","last_peered":"2023-10-06T01:42:03.656960+0000","last_clean":"2023-10-06T01:42:03.656960+0000","last_became_active":"2023-10-06T01:42:02.649568+0000","last_became_peered":"2023-10-06T01:42:02.649568+0000","last_unstale":"2023-10-06T01:42:03.656960+0000","last_undegraded":"2023-10-06T01:42:03.656960+0000","last_fullsized":"2023-10-06T01:42:03.656960+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:28:07.168502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.657093+0000","last_change":"2023-10-06T01:42:03.652468+0000","last_active":"2023-10-06T01:42:03.657093+0000","last_peered":"2023-10-06T01:42:03.657093+0000","last_clean":"2023-10-06T01:42:03.657093+0000","last_became_active":"2023-10-06T01:42:03.652170+0000","last_became_peered":"2023-10-06T01:42:03.652170+0000","last_unstale":"2023-10-06T01:42:03.657093+0000","last_undegraded":"2023-10-06T01:42:03.657093+0000","last_fullsized":"2023-10-06T01:42:03.657093+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:50:12.712279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-06T01:42:02.654149+0000","last_change":"2023-10-06T01:42:02.649181+0000","last_active":"2023-10-06T01:42:02.654149+0000","last_peered":"2023-10-06T01:42:02.654149+0000","last_clean":"2023-10-06T01:42:02.654149+0000","last_became_active":"2023-10-06T01:42:02.648911+0000","last_became_peered":"2023-10-06T01:42:02.648911+0000","last_unstale":"2023-10-06T01:42:02.654149+0000","last_undegraded":"2023-10-06T01:42:02.654149+0000","last_fullsized":"2023-10-06T01:42:02.654149+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:15:03.482499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.658023+0000","last_change":"2023-10-06T01:42:03.649461+0000","last_active":"2023-10-06T01:42:03.658023+0000","last_peered":"2023-10-06T01:42:03.658023+0000","last_clean":"2023-10-06T01:42:03.658023+0000","last_became_active":"2023-10-06T01:42:03.649160+0000","last_became_peered":"2023-10-06T01:42:03.649160+0000","last_unstale":"2023-10-06T01:42:03.658023+0000","last_undegraded":"2023-10-06T01:42:03.658023+0000","last_fullsized":"2023-10-06T01:42:03.658023+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:10:24.478901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"30'2","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.659017+0000","last_change":"2023-10-06T01:42:02.645240+0000","last_active":"2023-10-06T01:42:03.659017+0000","last_peered":"2023-10-06T01:42:03.659017+0000","last_clean":"2023-10-06T01:42:03.659017+0000","last_became_active":"2023-10-06T01:42:02.644974+0000","last_became_peered":"2023-10-06T01:42:02.644974+0000","last_unstale":"2023-10-06T01:42:03.659017+0000","last_undegraded":"2023-10-06T01:42:03.659017+0000","last_fullsized":"2023-10-06T01:42:03.659017+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:59:48.617985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.662472+0000","last_change":"2023-10-06T01:42:03.654680+0000","last_active":"2023-10-06T01:42:03.662472+0000","last_peered":"2023-10-06T01:42:03.662472+0000","last_clean":"2023-10-06T01:42:03.662472+0000","last_became_active":"2023-10-06T01:42:03.654382+0000","last_became_peered":"2023-10-06T01:42:03.654382+0000","last_unstale":"2023-10-06T01:42:03.662472+0000","last_undegraded":"2023-10-06T01:42:03.662472+0000","last_fullsized":"2023-10-06T01:42:03.662472+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:50:55.550267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.642565+0000","last_change":"2023-10-06T01:42:02.650108+0000","last_active":"2023-10-06T01:42:03.642565+0000","last_peered":"2023-10-06T01:42:03.642565+0000","last_clean":"2023-10-06T01:42:03.642565+0000","last_became_active":"2023-10-06T01:42:02.649810+0000","last_became_peered":"2023-10-06T01:42:02.649810+0000","last_unstale":"2023-10-06T01:42:03.642565+0000","last_undegraded":"2023-10-06T01:42:03.642565+0000","last_fullsized":"2023-10-06T01:42:03.642565+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:50:43.763733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.656709+0000","last_change":"2023-10-06T01:42:03.654313+0000","last_active":"2023-10-06T01:42:03.656709+0000","last_peered":"2023-10-06T01:42:03.656709+0000","last_clean":"2023-10-06T01:42:03.656709+0000","last_became_active":"2023-10-06T01:42:03.654029+0000","last_became_peered":"2023-10-06T01:42:03.654029+0000","last_unstale":"2023-10-06T01:42:03.656709+0000","last_undegraded":"2023-10-06T01:42:03.656709+0000","last_fullsized":"2023-10-06T01:42:03.656709+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:56:54.630363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.639372+0000","last_change":"2023-10-06T01:42:02.649943+0000","last_active":"2023-10-06T01:42:03.639372+0000","last_peered":"2023-10-06T01:42:03.639372+0000","last_clean":"2023-10-06T01:42:03.639372+0000","last_became_active":"2023-10-06T01:42:02.649657+0000","last_became_peered":"2023-10-06T01:42:02.649657+0000","last_unstale":"2023-10-06T01:42:03.639372+0000","last_undegraded":"2023-10-06T01:42:03.639372+0000","last_fullsized":"2023-10-06T01:42:03.639372+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:54:48.674855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":8,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.656556+0000","last_change":"2023-10-06T01:42:03.656556+0000","last_active":"2023-10-06T01:42:03.656556+0000","last_peered":"2023-10-06T01:42:03.656556+0000","last_clean":"2023-10-06T01:42:03.656556+0000","last_became_active":"2023-10-06T01:42:03.656233+0000","last_became_peered":"2023-10-06T01:42:03.656233+0000","last_unstale":"2023-10-06T01:42:03.656556+0000","last_undegraded":"2023-10-06T01:42:03.656556+0000","last_fullsized":"2023-10-06T01:42:03.656556+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:56:54.630363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.642425+0000","last_change":"2023-10-06T01:42:02.644470+0000","last_active":"2023-10-06T01:42:03.642425+0000","last_peered":"2023-10-06T01:42:03.642425+0000","last_clean":"2023-10-06T01:42:03.642425+0000","last_became_active":"2023-10-06T01:42:02.644180+0000","last_became_peered":"2023-10-06T01:42:02.644180+0000","last_unstale":"2023-10-06T01:42:03.642425+0000","last_undegraded":"2023-10-06T01:42:03.642425+0000","last_fullsized":"2023-10-06T01:42:03.642425+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:12:00.299594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.665317+0000","last_change":"2023-10-06T01:42:03.661293+0000","last_active":"2023-10-06T01:42:03.665317+0000","last_peered":"2023-10-06T01:42:03.665317+0000","last_clean":"2023-10-06T01:42:03.665317+0000","last_became_active":"2023-10-06T01:42:03.660488+0000","last_became_peered":"2023-10-06T01:42:03.660488+0000","last_unstale":"2023-10-06T01:42:03.665317+0000","last_undegraded":"2023-10-06T01:42:03.665317+0000","last_fullsized":"2023-10-06T01:42:03.665317+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:55:56.347224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.644673+0000","last_change":"2023-10-06T01:42:02.643814+0000","last_active":"2023-10-06T01:42:03.644673+0000","last_peered":"2023-10-06T01:42:03.644673+0000","last_clean":"2023-10-06T01:42:03.644673+0000","last_became_active":"2023-10-06T01:42:02.643600+0000","last_became_peered":"2023-10-06T01:42:02.643600+0000","last_unstale":"2023-10-06T01:42:03.644673+0000","last_undegraded":"2023-10-06T01:42:03.644673+0000","last_fullsized":"2023-10-06T01:42:03.644673+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:37:14.256974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":23,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.648899+0000","last_change":"2023-10-06T01:41:07.454295+0000","last_active":"2023-10-06T01:42:03.648899+0000","last_peered":"2023-10-06T01:42:03.648899+0000","last_clean":"2023-10-06T01:42:03.648899+0000","last_became_active":"2023-10-06T01:41:07.454034+0000","last_became_peered":"2023-10-06T01:41:07.454034+0000","last_unstale":"2023-10-06T01:42:03.648899+0000","last_undegraded":"2023-10-06T01:42:03.648899+0000","last_fullsized":"2023-10-06T01:42:03.648899+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:32:58.914183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.660860+0000","last_change":"2023-10-06T01:42:03.657085+0000","last_active":"2023-10-06T01:42:03.660860+0000","last_peered":"2023-10-06T01:42:03.660860+0000","last_clean":"2023-10-06T01:42:03.660860+0000","last_became_active":"2023-10-06T01:42:03.654852+0000","last_became_peered":"2023-10-06T01:42:03.654852+0000","last_unstale":"2023-10-06T01:42:03.660860+0000","last_undegraded":"2023-10-06T01:42:03.660860+0000","last_fullsized":"2023-10-06T01:42:03.660860+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:18:14.223068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-06T01:42:02.648150+0000","last_change":"2023-10-06T01:42:02.644525+0000","last_active":"2023-10-06T01:42:02.648150+0000","last_peered":"2023-10-06T01:42:02.648150+0000","last_clean":"2023-10-06T01:42:02.648150+0000","last_became_active":"2023-10-06T01:42:02.644226+0000","last_became_peered":"2023-10-06T01:42:02.644226+0000","last_unstale":"2023-10-06T01:42:02.648150+0000","last_undegraded":"2023-10-06T01:42:02.648150+0000","last_fullsized":"2023-10-06T01:42:02.648150+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:40:35.423082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-06T01:42:02.632076+0000","last_change":"2023-10-06T01:41:07.261955+0000","last_active":"2023-10-06T01:42:02.632076+0000","last_peered":"2023-10-06T01:42:02.632076+0000","last_clean":"2023-10-06T01:42:02.632076+0000","last_became_active":"2023-10-06T01:41:07.261692+0000","last_became_peered":"2023-10-06T01:41:07.261692+0000","last_unstale":"2023-10-06T01:42:02.632076+0000","last_undegraded":"2023-10-06T01:42:02.632076+0000","last_fullsized":"2023-10-06T01:42:02.632076+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":8,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.654638+0000","last_change":"2023-10-06T01:42:03.654638+0000","last_active":"2023-10-06T01:42:03.654638+0000","last_peered":"2023-10-06T01:42:03.654638+0000","last_clean":"2023-10-06T01:42:03.654638+0000","last_became_active":"2023-10-06T01:42:03.654329+0000","last_became_peered":"2023-10-06T01:42:03.654329+0000","last_unstale":"2023-10-06T01:42:03.654638+0000","last_undegraded":"2023-10-06T01:42:03.654638+0000","last_fullsized":"2023-10-06T01:42:03.654638+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:07:16.583330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"30'2","reported_seq":12,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.653885+0000","last_change":"2023-10-06T01:42:02.648894+0000","last_active":"2023-10-06T01:42:03.653885+0000","last_peered":"2023-10-06T01:42:03.653885+0000","last_clean":"2023-10-06T01:42:03.653885+0000","last_became_active":"2023-10-06T01:42:02.648494+0000","last_became_peered":"2023-10-06T01:42:02.648494+0000","last_unstale":"2023-10-06T01:42:03.653885+0000","last_undegraded":"2023-10-06T01:42:03.653885+0000","last_fullsized":"2023-10-06T01:42:03.653885+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T03:40:35.423082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":20,"reported_epoch":29,"state":"active+clean","last_fresh":"2023-10-06T01:42:02.643188+0000","last_change":"2023-10-06T01:41:07.258362+0000","last_active":"2023-10-06T01:42:02.643188+0000","last_peered":"2023-10-06T01:42:02.643188+0000","last_clean":"2023-10-06T01:42:02.643188+0000","last_became_active":"2023-10-06T01:41:07.258055+0000","last_became_peered":"2023-10-06T01:41:07.258055+0000","last_unstale":"2023-10-06T01:42:02.643188+0000","last_undegraded":"2023-10-06T01:42:02.643188+0000","last_fullsized":"2023-10-06T01:42:02.643188+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"16'117","reported_seq":171,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.647279+0000","last_change":"2023-10-06T01:41:07.261164+0000","last_active":"2023-10-06T01:42:03.647279+0000","last_peered":"2023-10-06T01:42:03.647279+0000","last_clean":"2023-10-06T01:42:03.647279+0000","last_became_active":"2023-10-06T01:41:07.260896+0000","last_became_peered":"2023-10-06T01:41:07.260896+0000","last_unstale":"2023-10-06T01:42:03.647279+0000","last_undegraded":"2023-10-06T01:42:03.647279+0000","last_fullsized":"2023-10-06T01:42:03.647279+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:05:22.934401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.660972+0000","last_change":"2023-10-06T01:42:03.657656+0000","last_active":"2023-10-06T01:42:03.660972+0000","last_peered":"2023-10-06T01:42:03.660972+0000","last_clean":"2023-10-06T01:42:03.660972+0000","last_became_active":"2023-10-06T01:42:03.657433+0000","last_became_peered":"2023-10-06T01:42:03.657433+0000","last_unstale":"2023-10-06T01:42:03.660972+0000","last_undegraded":"2023-10-06T01:42:03.660972+0000","last_fullsized":"2023-10-06T01:42:03.660972+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:06:21.581530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"31'1","reported_seq":11,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-06T01:42:04.652334+0000","last_change":"2023-10-06T01:42:02.650129+0000","last_active":"2023-10-06T01:42:04.652334+0000","last_peered":"2023-10-06T01:42:04.652334+0000","last_clean":"2023-10-06T01:42:04.652334+0000","last_became_active":"2023-10-06T01:42:02.649835+0000","last_became_peered":"2023-10-06T01:42:02.649835+0000","last_unstale":"2023-10-06T01:42:04.652334+0000","last_undegraded":"2023-10-06T01:42:04.652334+0000","last_fullsized":"2023-10-06T01:42:04.652334+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:19:19.661213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":21,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.639329+0000","last_change":"2023-10-06T01:41:07.456511+0000","last_active":"2023-10-06T01:42:03.639329+0000","last_peered":"2023-10-06T01:42:03.639329+0000","last_clean":"2023-10-06T01:42:03.639329+0000","last_became_active":"2023-10-06T01:41:07.456024+0000","last_became_peered":"2023-10-06T01:41:07.456024+0000","last_unstale":"2023-10-06T01:42:03.639329+0000","last_undegraded":"2023-10-06T01:42:03.639329+0000","last_fullsized":"2023-10-06T01:42:03.639329+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:51:02.949231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.656677+0000","last_change":"2023-10-06T01:42:03.654421+0000","last_active":"2023-10-06T01:42:03.656677+0000","last_peered":"2023-10-06T01:42:03.656677+0000","last_clean":"2023-10-06T01:42:03.656677+0000","last_became_active":"2023-10-06T01:42:03.654144+0000","last_became_peered":"2023-10-06T01:42:03.654144+0000","last_unstale":"2023-10-06T01:42:03.656677+0000","last_undegraded":"2023-10-06T01:42:03.656677+0000","last_fullsized":"2023-10-06T01:42:03.656677+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:06:21.581530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.643884+0000","last_change":"2023-10-06T01:42:02.648994+0000","last_active":"2023-10-06T01:42:03.643884+0000","last_peered":"2023-10-06T01:42:03.643884+0000","last_clean":"2023-10-06T01:42:03.643884+0000","last_became_active":"2023-10-06T01:42:02.648564+0000","last_became_peered":"2023-10-06T01:42:02.648564+0000","last_unstale":"2023-10-06T01:42:03.643884+0000","last_undegraded":"2023-10-06T01:42:03.643884+0000","last_fullsized":"2023-10-06T01:42:03.643884+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:36:37.018597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":21,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.647187+0000","last_change":"2023-10-06T01:41:07.456273+0000","last_active":"2023-10-06T01:42:03.647187+0000","last_peered":"2023-10-06T01:42:03.647187+0000","last_clean":"2023-10-06T01:42:03.647187+0000","last_became_active":"2023-10-06T01:41:07.456000+0000","last_became_peered":"2023-10-06T01:41:07.456000+0000","last_unstale":"2023-10-06T01:42:03.647187+0000","last_undegraded":"2023-10-06T01:42:03.647187+0000","last_fullsized":"2023-10-06T01:42:03.647187+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:01:42.633388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":8,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.663945+0000","last_change":"2023-10-06T01:42:03.663945+0000","last_active":"2023-10-06T01:42:03.663945+0000","last_peered":"2023-10-06T01:42:03.663945+0000","last_clean":"2023-10-06T01:42:03.663945+0000","last_became_active":"2023-10-06T01:42:03.663698+0000","last_became_peered":"2023-10-06T01:42:03.663698+0000","last_unstale":"2023-10-06T01:42:03.663945+0000","last_undegraded":"2023-10-06T01:42:03.663945+0000","last_fullsized":"2023-10-06T01:42:03.663945+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:34:59.205863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.636149+0000","last_change":"2023-10-06T01:42:02.650860+0000","last_active":"2023-10-06T01:42:03.636149+0000","last_peered":"2023-10-06T01:42:03.636149+0000","last_clean":"2023-10-06T01:42:03.636149+0000","last_became_active":"2023-10-06T01:42:02.650567+0000","last_became_peered":"2023-10-06T01:42:02.650567+0000","last_unstale":"2023-10-06T01:42:03.636149+0000","last_undegraded":"2023-10-06T01:42:03.636149+0000","last_fullsized":"2023-10-06T01:42:03.636149+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:48:55.898734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":22,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.643858+0000","last_change":"2023-10-06T01:41:07.456688+0000","last_active":"2023-10-06T01:42:03.643858+0000","last_peered":"2023-10-06T01:42:03.643858+0000","last_clean":"2023-10-06T01:42:03.643858+0000","last_became_active":"2023-10-06T01:41:07.456304+0000","last_became_peered":"2023-10-06T01:41:07.456304+0000","last_unstale":"2023-10-06T01:42:03.643858+0000","last_undegraded":"2023-10-06T01:42:03.643858+0000","last_fullsized":"2023-10-06T01:42:03.643858+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":8,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.641989+0000","last_change":"2023-10-06T01:42:03.641989+0000","last_active":"2023-10-06T01:42:03.641989+0000","last_peered":"2023-10-06T01:42:03.641989+0000","last_clean":"2023-10-06T01:42:03.641989+0000","last_became_active":"2023-10-06T01:42:03.641714+0000","last_became_peered":"2023-10-06T01:42:03.641714+0000","last_unstale":"2023-10-06T01:42:03.641989+0000","last_undegraded":"2023-10-06T01:42:03.641989+0000","last_fullsized":"2023-10-06T01:42:03.641989+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:24:18.930221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.637250+0000","last_change":"2023-10-06T01:42:02.648007+0000","last_active":"2023-10-06T01:42:03.637250+0000","last_peered":"2023-10-06T01:42:03.637250+0000","last_clean":"2023-10-06T01:42:03.637250+0000","last_became_active":"2023-10-06T01:42:02.647751+0000","last_became_peered":"2023-10-06T01:42:02.647751+0000","last_unstale":"2023-10-06T01:42:03.637250+0000","last_undegraded":"2023-10-06T01:42:03.637250+0000","last_fullsized":"2023-10-06T01:42:03.637250+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T13:41:03.238103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":23,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.638089+0000","last_change":"2023-10-06T01:41:07.456388+0000","last_active":"2023-10-06T01:42:03.638089+0000","last_peered":"2023-10-06T01:42:03.638089+0000","last_clean":"2023-10-06T01:42:03.638089+0000","last_became_active":"2023-10-06T01:41:07.456045+0000","last_became_peered":"2023-10-06T01:41:07.456045+0000","last_unstale":"2023-10-06T01:42:03.638089+0000","last_undegraded":"2023-10-06T01:42:03.638089+0000","last_fullsized":"2023-10-06T01:42:03.638089+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":9,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.662503+0000","last_change":"2023-10-06T01:42:03.650881+0000","last_active":"2023-10-06T01:42:03.662503+0000","last_peered":"2023-10-06T01:42:03.662503+0000","last_clean":"2023-10-06T01:42:03.662503+0000","last_became_active":"2023-10-06T01:42:03.650592+0000","last_became_peered":"2023-10-06T01:42:03.650592+0000","last_unstale":"2023-10-06T01:42:03.662503+0000","last_undegraded":"2023-10-06T01:42:03.662503+0000","last_fullsized":"2023-10-06T01:42:03.662503+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":29,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:02.625951+0000","last_clean_scrub_stamp":"2023-10-06T01:42:02.625951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:18:14.223068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":10,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.635993+0000","last_change":"2023-10-06T01:42:02.650163+0000","last_active":"2023-10-06T01:42:03.635993+0000","last_peered":"2023-10-06T01:42:03.635993+0000","last_clean":"2023-10-06T01:42:03.635993+0000","last_became_active":"2023-10-06T01:42:02.649353+0000","last_became_peered":"2023-10-06T01:42:02.649353+0000","last_unstale":"2023-10-06T01:42:03.635993+0000","last_undegraded":"2023-10-06T01:42:03.635993+0000","last_fullsized":"2023-10-06T01:42:03.635993+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":29,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:01.619802+0000","last_clean_scrub_stamp":"2023-10-06T01:42:01.619802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T11:04:20.088699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":23,"reported_epoch":30,"state":"active+clean","last_fresh":"2023-10-06T01:42:03.638208+0000","last_change":"2023-10-06T01:41:07.258162+0000","last_active":"2023-10-06T01:42:03.638208+0000","last_peered":"2023-10-06T01:42:03.638208+0000","last_clean":"2023-10-06T01:42:03.638208+0000","last_became_active":"2023-10-06T01:41:07.257795+0000","last_became_peered":"2023-10-06T01:41:07.257795+0000","last_unstale":"2023-10-06T01:42:03.638208+0000","last_undegraded":"2023-10-06T01:42:03.638208+0000","last_fullsized":"2023-10-06T01:42:03.638208+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:05:22.934401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2306,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298212,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073628,"statfs":{"total":96636764160,"available":96331395072,"internally_reserved":0,"allocated":946176,"data_stored":726468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68600000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]}]},{"osd":6,"up_from":13,"seq":55834574866,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297628,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074212,"statfs":{"total":96636764160,"available":96331993088,"internally_reserved":0,"allocated":348160,"data_stored":140336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.316}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67500000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574864,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298200,"kb_used_data":912,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073640,"statfs":{"total":96636764160,"available":96331407360,"internally_reserved":0,"allocated":933888,"data_stored":724952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.317}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40400000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]}]},{"osd":0,"up_from":13,"seq":55834574864,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297616,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074224,"statfs":{"total":96636764160,"available":96332005376,"internally_reserved":0,"allocated":335872,"data_stored":133175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54100000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42099999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574865,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297632,"kb_used_data":344,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074208,"statfs":{"total":96636764160,"available":96331988992,"internally_reserved":0,"allocated":352256,"data_stored":141205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.253}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]}]},{"osd":3,"up_from":13,"seq":55834574864,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297628,"kb_used_data":340,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074212,"statfs":{"total":96636764160,"available":96331993088,"internally_reserved":0,"allocated":348160,"data_stored":134711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54700000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]}]},{"osd":4,"up_from":13,"seq":55834574865,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297636,"kb_used_data":348,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074204,"statfs":{"total":96636764160,"available":96331984896,"internally_reserved":0,"allocated":356352,"data_stored":141711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46999999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39100000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.377}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574863,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297632,"kb_used_data":344,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074208,"statfs":{"total":96636764160,"available":96331988992,"internally_reserved":0,"allocated":352256,"data_stored":139894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.747},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.335,"5min":1.335,"15min":1.335},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":2.4849999999999999,"5min":2.4849999999999999,"15min":2.4849999999999999},"last":0.68600000000000005}]},{"osd":2,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.8280000000000001,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":0.71299999999999997}]},{"osd":3,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.9830000000000001,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.66200000000000003}]},{"osd":4,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.32800000000000001}]},{"osd":6,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.57299999999999995}]},{"osd":7,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.59499999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-06T01:42:08.126 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-06T01:42:08.126 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-06T01:42:08.286 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.284+0000 7f0db3e80700 1 Processor -- start 2023-10-06T01:42:08.286 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.284+0000 7f0db3e80700 1 -- start start 2023-10-06T01:42:08.289 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.285+0000 7f0db3e80700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f0dac159890 0x7f0dac0b0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:08.289 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.285+0000 7f0db3e80700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dac0cd860 con 0x7f0dac0b1be0 2023-10-06T01:42:08.289 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.285+0000 7f0db3e80700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dac0ccc50 con 0x7f0dac0cdcf0 2023-10-06T01:42:08.290 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.285+0000 7f0db3e80700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dac0ce190 con 0x7f0dac159890 2023-10-06T01:42:08.290 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.285+0000 7f0db241d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0dac0b1be0 0x7f0dac159c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52098/0 (socket says 172.21.15.27:52098) 2023-10-06T01:42:08.290 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.285+0000 7f0db241d700 1 -- 172.21.15.27:0/1459473509 learned_addr learned my addr 172.21.15.27:0/1459473509 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:08.290 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.286+0000 7f0db1c1c700 1 --2- 172.21.15.27:0/1459473509 >> v2:172.21.15.124:3300/0 conn(0x7f0dac159890 0x7f0dac0b0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:08.290 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.286+0000 7f0db0c1a700 1 -- 172.21.15.27:0/1459473509 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1392866188 0 0) 0x7f0dac0cd860 con 0x7f0dac0b1be0 2023-10-06T01:42:08.290 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.286+0000 7f0db0c1a700 1 -- 172.21.15.27:0/1459473509 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d94003570 con 0x7f0dac0b1be0 2023-10-06T01:42:08.290 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.286+0000 7f0db0c1a700 1 -- 172.21.15.27:0/1459473509 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 806080053 0 0) 0x7f0d94003570 con 0x7f0dac0b1be0 2023-10-06T01:42:08.290 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.286+0000 7f0db0c1a700 1 -- 172.21.15.27:0/1459473509 >> v2:172.21.15.124:3300/0 conn(0x7f0dac159890 msgr2=0x7f0dac0b0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:08.290 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.286+0000 7f0db0c1a700 1 --2- 172.21.15.27:0/1459473509 >> v2:172.21.15.124:3300/0 conn(0x7f0dac159890 0x7f0dac0b0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:08.290 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.286+0000 7f0db0c1a700 1 -- 172.21.15.27:0/1459473509 >> v1:172.21.15.124:6789/0 conn(0x7f0dac0cdcf0 legacy=0x7f0dac0b14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:08.290 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.286+0000 7f0db0c1a700 1 -- 172.21.15.27:0/1459473509 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dac0ccf30 con 0x7f0dac0b1be0 2023-10-06T01:42:08.291 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.286+0000 7f0db0c1a700 1 -- 172.21.15.27:0/1459473509 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2619078944 0 0) 0x7f0d9c0017f0 con 0x7f0dac0b1be0 2023-10-06T01:42:08.291 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.286+0000 7f0db0c1a700 1 -- 172.21.15.27:0/1459473509 >> v1:172.21.15.27:6789/0 conn(0x7f0dac0b1be0 legacy=0x7f0dac159c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:08.291 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.286+0000 7f0db0c1a700 1 --2- 172.21.15.27:0/1459473509 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0d94003a60 0x7f0d94003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:08.291 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.286+0000 7f0db0c1a700 1 --2- 172.21.15.27:0/1459473509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d94004810 0x7f0d94004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:08.291 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.286+0000 7f0db0c1a700 1 --2- 172.21.15.27:0/1459473509 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0d940051c0 0x7f0d9400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:08.291 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.286+0000 7f0db0c1a700 1 -- 172.21.15.27:0/1459473509 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9400c130 con 0x7f0d94004810 2023-10-06T01:42:08.291 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.286+0000 7f0db0c1a700 1 -- 172.21.15.27:0/1459473509 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9400c2a0 con 0x7f0d94003a60 2023-10-06T01:42:08.291 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.286+0000 7f0db0c1a700 1 -- 172.21.15.27:0/1459473509 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9400c460 con 0x7f0d940051c0 2023-10-06T01:42:08.291 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.286+0000 7f0db141b700 1 --2- 172.21.15.27:0/1459473509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d94004810 0x7f0d94004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:08.291 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.286+0000 7f0db1c1c700 1 --2- 172.21.15.27:0/1459473509 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0d94003a60 0x7f0d94003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:08.291 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.286+0000 7f0db241d700 1 --2- 172.21.15.27:0/1459473509 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0d940051c0 0x7f0d9400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:08.292 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.286+0000 7f0db141b700 1 -- 172.21.15.27:0/1459473509 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0d940051c0 msgr2=0x7f0d9400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:08.292 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.286+0000 7f0db141b700 1 --2- 172.21.15.27:0/1459473509 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0d940051c0 0x7f0d9400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:08.292 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.286+0000 7f0db141b700 1 -- 172.21.15.27:0/1459473509 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0d94003a60 msgr2=0x7f0d94003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:08.292 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.286+0000 7f0db141b700 1 --2- 172.21.15.27:0/1459473509 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0d94003a60 0x7f0d94003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:08.292 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.286+0000 7f0db241d700 1 -- 172.21.15.27:0/1459473509 reap_dead start 2023-10-06T01:42:08.292 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.286+0000 7f0db141b700 1 -- 172.21.15.27:0/1459473509 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d94003570 con 0x7f0d94004810 2023-10-06T01:42:08.292 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.287+0000 7f0db141b700 1 --2- 172.21.15.27:0/1459473509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d94004810 0x7f0d94004c80 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0da00018b0 tx=0x7f0da0007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=645558552ce0b6bb server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:08.292 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.287+0000 7f0db0c1a700 1 -- 172.21.15.27:0/1459473509 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da0012420 con 0x7f0d94004810 2023-10-06T01:42:08.292 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.287+0000 7f0db0c1a700 1 -- 172.21.15.27:0/1459473509 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0da00125a0 con 0x7f0d94004810 2023-10-06T01:42:08.292 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.287+0000 7f0db0c1a700 1 -- 172.21.15.27:0/1459473509 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da0012960 con 0x7f0d94004810 2023-10-06T01:42:08.292 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.287+0000 7f0db3e80700 1 -- 172.21.15.27:0/1459473509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d94004810 msgr2=0x7f0d94004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:08.292 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.287+0000 7f0db3e80700 1 --2- 172.21.15.27:0/1459473509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d94004810 0x7f0d94004c80 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0da00018b0 tx=0x7f0da0007d30 comp rx=0 tx=0).stop 2023-10-06T01:42:08.293 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.287+0000 7f0db3e80700 1 -- 172.21.15.27:0/1459473509 shutdown_connections 2023-10-06T01:42:08.293 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.287+0000 7f0db3e80700 1 --2- 172.21.15.27:0/1459473509 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0d94004810 0x7f0d94004c80 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:08.293 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.287+0000 7f0db3e80700 1 -- 172.21.15.27:0/1459473509 >> 172.21.15.27:0/1459473509 conn(0x7f0dac099990 msgr2=0x7f0dac0b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:08.293 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.287+0000 7f0db3e80700 1 -- 172.21.15.27:0/1459473509 shutdown_connections 2023-10-06T01:42:08.293 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.287+0000 7f0db3e80700 1 -- 172.21.15.27:0/1459473509 wait complete. 2023-10-06T01:42:08.295 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.289+0000 7f0db3e80700 1 Processor -- start 2023-10-06T01:42:08.295 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.289+0000 7f0db3e80700 1 -- start start 2023-10-06T01:42:08.296 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.289+0000 7f0db3e80700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0d94004810 0x7f0dac16d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:08.296 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.289+0000 7f0db3e80700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0dac159c60 0x7f0dac0d0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:08.296 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.289+0000 7f0db3e80700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dac0d0d80 0x7f0dac155790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:08.296 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.289+0000 7f0db3e80700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dac0aab70 con 0x7f0dac0d0d80 2023-10-06T01:42:08.296 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.289+0000 7f0db3e80700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dac16d8b0 con 0x7f0dac159c60 2023-10-06T01:42:08.296 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.289+0000 7f0db3e80700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dac15a7e0 con 0x7f0d94004810 2023-10-06T01:42:08.296 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.289+0000 7f0db241d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dac0d0d80 0x7f0dac155790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:08.296 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.289+0000 7f0db241d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dac0d0d80 0x7f0dac155790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37476/0 (socket says 172.21.15.27:37476) 2023-10-06T01:42:08.296 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.289+0000 7f0db241d700 1 -- 172.21.15.27:0/3155574310 learned_addr learned my addr 172.21.15.27:0/3155574310 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:08.297 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.290+0000 7f0db141b700 1 --2- 172.21.15.27:0/3155574310 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0dac159c60 0x7f0dac0d0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:08.297 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.290+0000 7f0db1c1c700 1 --2- 172.21.15.27:0/3155574310 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0d94004810 0x7f0dac16d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:08.297 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.290+0000 7f0db241d700 1 -- 172.21.15.27:0/3155574310 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0d94004810 msgr2=0x7f0dac16d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:08.297 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.290+0000 7f0db241d700 1 --2- 172.21.15.27:0/3155574310 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0d94004810 0x7f0dac16d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:08.297 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.290+0000 7f0db241d700 1 -- 172.21.15.27:0/3155574310 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0dac159c60 msgr2=0x7f0dac0d0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:08.298 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.290+0000 7f0db241d700 1 --2- 172.21.15.27:0/3155574310 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0dac159c60 0x7f0dac0d0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:08.298 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.290+0000 7f0db241d700 1 -- 172.21.15.27:0/3155574310 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dac16dae0 con 0x7f0dac0d0d80 2023-10-06T01:42:08.298 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.290+0000 7f0db241d700 1 --2- 172.21.15.27:0/3155574310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dac0d0d80 0x7f0dac155790 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c000f80 tx=0x7f0d9c001ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:08.298 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.290+0000 7f0da9ffb700 1 -- 172.21.15.27:0/3155574310 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d9c00eb10 con 0x7f0dac0d0d80 2023-10-06T01:42:08.298 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.290+0000 7f0da9ffb700 1 -- 172.21.15.27:0/3155574310 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d9c00ec70 con 0x7f0dac0d0d80 2023-10-06T01:42:08.299 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.290+0000 7f0db3e80700 1 -- 172.21.15.27:0/3155574310 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dac16ddc0 con 0x7f0dac0d0d80 2023-10-06T01:42:08.299 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.290+0000 7f0da9ffb700 1 -- 172.21.15.27:0/3155574310 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d9c013420 con 0x7f0dac0d0d80 2023-10-06T01:42:08.299 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.290+0000 7f0db3e80700 1 -- 172.21.15.27:0/3155574310 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dac155cd0 con 0x7f0dac0d0d80 2023-10-06T01:42:08.299 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.291+0000 7f0db3e80700 1 -- 172.21.15.27:0/3155574310 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dac16e2e0 con 0x7f0dac0d0d80 2023-10-06T01:42:08.300 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.292+0000 7f0da9ffb700 1 -- 172.21.15.27:0/3155574310 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0d9c0137b0 con 0x7f0dac0d0d80 2023-10-06T01:42:08.300 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.292+0000 7f0da9ffb700 1 --2- 172.21.15.27:0/3155574310 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0d98066f30 0x7f0d980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:08.300 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.292+0000 7f0da9ffb700 1 -- 172.21.15.27:0/3155574310 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f0d9c084bd0 con 0x7f0dac0d0d80 2023-10-06T01:42:08.301 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.292+0000 7f0db1c1c700 1 --2- 172.21.15.27:0/3155574310 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0d98066f30 0x7f0d980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:08.301 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.293+0000 7f0db1c1c700 1 --2- 172.21.15.27:0/3155574310 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0d98066f30 0x7f0d980693e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0da4006ea0 tx=0x7f0da4009120 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:08.302 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.296+0000 7f0da9ffb700 1 -- 172.21.15.27:0/3155574310 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0dac16e2e0 con 0x7f0dac0d0d80 2023-10-06T01:42:08.573 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.572+0000 7f0db3e80700 1 -- 172.21.15.27:0/3155574310 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f0dac15a5b0 con 0x7f0dac0d0d80 2023-10-06T01:42:08.576 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.575+0000 7f0da9ffb700 1 -- 172.21.15.27:0/3155574310 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+0 (secure 0 0 0) 0x7f0dac15a5b0 con 0x7f0dac0d0d80 2023-10-06T01:42:08.578 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.577+0000 7f0db3e80700 1 -- 172.21.15.27:0/3155574310 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0d98066f30 msgr2=0x7f0d980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:08.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.577+0000 7f0db3e80700 1 --2- 172.21.15.27:0/3155574310 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0d98066f30 0x7f0d980693e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f0da4006ea0 tx=0x7f0da4009120 comp rx=0 tx=0).stop 2023-10-06T01:42:08.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.577+0000 7f0db3e80700 1 -- 172.21.15.27:0/3155574310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dac0d0d80 msgr2=0x7f0dac155790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:08.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.577+0000 7f0db3e80700 1 --2- 172.21.15.27:0/3155574310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dac0d0d80 0x7f0dac155790 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c000f80 tx=0x7f0d9c001ff0 comp rx=0 tx=0).stop 2023-10-06T01:42:08.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.577+0000 7f0db3e80700 1 -- 172.21.15.27:0/3155574310 shutdown_connections 2023-10-06T01:42:08.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.577+0000 7f0db3e80700 1 --2- 172.21.15.27:0/3155574310 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0d98066f30 0x7f0d980693e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:08.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.577+0000 7f0db3e80700 1 --2- 172.21.15.27:0/3155574310 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0d94004810 0x7f0dac16d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:08.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.577+0000 7f0db3e80700 1 --2- 172.21.15.27:0/3155574310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0dac0d0d80 0x7f0dac155790 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:08.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.577+0000 7f0db3e80700 1 --2- 172.21.15.27:0/3155574310 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0dac159c60 0x7f0dac0d0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:08.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.577+0000 7f0db3e80700 1 -- 172.21.15.27:0/3155574310 >> 172.21.15.27:0/3155574310 conn(0x7f0dac099990 msgr2=0x7f0dac0679f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:08.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.577+0000 7f0db3e80700 1 -- 172.21.15.27:0/3155574310 shutdown_connections 2023-10-06T01:42:08.580 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.577+0000 7f0db3e80700 1 -- 172.21.15.27:0/3155574310 wait complete. 2023-10-06T01:42:08.580 INFO:teuthology.orchestra.run.smithi027.stderr:updated caps for client.0 2023-10-06T01:42:08.590 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T01:42:08.710 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.708+0000 7ff7b6297700 1 Processor -- start 2023-10-06T01:42:08.710 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.708+0000 7ff7b6297700 1 -- start start 2023-10-06T01:42:08.712 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.709+0000 7ff7b6297700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff7b01596c0 0x7ff7b015d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:08.712 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.709+0000 7ff7b6297700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff7b00b09a0 0x7ff7b00c8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:08.712 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.709+0000 7ff7b6297700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7b015e3a0 con 0x7ff7b015d960 2023-10-06T01:42:08.712 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.709+0000 7ff7b6297700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7b00cc470 con 0x7ff7b00b09a0 2023-10-06T01:42:08.713 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.709+0000 7ff7b6297700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b015dde0 con 0x7ff7b01596c0 2023-10-06T01:42:08.713 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.709+0000 7ff7af7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff7b00b09a0 0x7ff7b00c8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:08.713 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.709+0000 7ff7af7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff7b00b09a0 0x7ff7b00c8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37478/0 (socket says 172.21.15.27:37478) 2023-10-06T01:42:08.713 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.709+0000 7ff7af7fe700 1 -- 172.21.15.27:0/4221423547 learned_addr learned my addr 172.21.15.27:0/4221423547 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:08.713 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.709+0000 7ff7b4834700 1 --1- 172.21.15.27:0/4221423547 >> v1:172.21.15.27:6789/0 conn(0x7ff7b015d960 0x7ff7b0068fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52108/0 (socket says 172.21.15.27:52108) 2023-10-06T01:42:08.713 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.709+0000 7ff7affff700 1 --2- 172.21.15.27:0/4221423547 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff7b01596c0 0x7ff7b015d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:08.713 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.710+0000 7ff7af7fe700 1 -- 172.21.15.27:0/4221423547 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff7b01596c0 msgr2=0x7ff7b015d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:08.713 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.710+0000 7ff7af7fe700 1 --2- 172.21.15.27:0/4221423547 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff7b01596c0 0x7ff7b015d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:08.713 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.710+0000 7ff7af7fe700 1 -- 172.21.15.27:0/4221423547 >> v1:172.21.15.27:6789/0 conn(0x7ff7b015d960 legacy=0x7ff7b0068fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:08.713 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.710+0000 7ff7af7fe700 1 -- 172.21.15.27:0/4221423547 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b015e0c0 con 0x7ff7b00b09a0 2023-10-06T01:42:08.713 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.710+0000 7ff7aeffd700 1 -- 172.21.15.27:0/4221423547 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3276887777 0 0) 0x7ff7b015e3a0 con 0x7ff7b015d960 2023-10-06T01:42:08.713 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.710+0000 7ff7af7fe700 1 --2- 172.21.15.27:0/4221423547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7b00b09a0 0x7ff7b00c8a40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff7a40099b0 tx=0x7ff7a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ab25e48fdcfffbb server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:08.714 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.710+0000 7ff7aeffd700 1 -- 172.21.15.27:0/4221423547 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a4009ea0 con 0x7ff7b00b09a0 2023-10-06T01:42:08.714 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.710+0000 7ff7aeffd700 1 -- 172.21.15.27:0/4221423547 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7a400b860 con 0x7ff7b00b09a0 2023-10-06T01:42:08.714 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.710+0000 7ff7aeffd700 1 -- 172.21.15.27:0/4221423547 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a400bc00 con 0x7ff7b00b09a0 2023-10-06T01:42:08.714 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.711+0000 7ff7b6297700 1 -- 172.21.15.27:0/4221423547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7b00b09a0 msgr2=0x7ff7b00c8a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:08.714 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.711+0000 7ff7b6297700 1 --2- 172.21.15.27:0/4221423547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7b00b09a0 0x7ff7b00c8a40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff7a40099b0 tx=0x7ff7a4009250 comp rx=0 tx=0).stop 2023-10-06T01:42:08.714 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.711+0000 7ff7b6297700 1 -- 172.21.15.27:0/4221423547 shutdown_connections 2023-10-06T01:42:08.714 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.711+0000 7ff7b6297700 1 --2- 172.21.15.27:0/4221423547 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff7b01596c0 0x7ff7b015d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:08.714 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.711+0000 7ff7b6297700 1 --2- 172.21.15.27:0/4221423547 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7b00b09a0 0x7ff7b00c8a40 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:08.714 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.711+0000 7ff7b6297700 1 -- 172.21.15.27:0/4221423547 >> 172.21.15.27:0/4221423547 conn(0x7ff7b0067630 msgr2=0x7ff7b0067a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:08.714 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.711+0000 7ff7b6297700 1 -- 172.21.15.27:0/4221423547 shutdown_connections 2023-10-06T01:42:08.714 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.711+0000 7ff7b6297700 1 -- 172.21.15.27:0/4221423547 wait complete. 2023-10-06T01:42:08.717 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.712+0000 7ff7b6297700 1 Processor -- start 2023-10-06T01:42:08.717 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.712+0000 7ff7b6297700 1 -- start start 2023-10-06T01:42:08.717 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.713+0000 7ff7b6297700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff7b01596c0 0x7ff7b0157a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:08.717 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.713+0000 7ff7b6297700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff7b015a4a0 0x7ff7b015a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:08.717 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.713+0000 7ff7b6297700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7b00c8750 0x7ff7b00c8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:08.718 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.713+0000 7ff7b6297700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b00afd00 con 0x7ff7b00c8750 2023-10-06T01:42:08.718 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.713+0000 7ff7b6297700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b0157fb0 con 0x7ff7b015a4a0 2023-10-06T01:42:08.718 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.713+0000 7ff7b6297700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b00c5910 con 0x7ff7b01596c0 2023-10-06T01:42:08.718 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.713+0000 7ff7b4834700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7b00c8750 0x7ff7b00c8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:08.718 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.713+0000 7ff7b4834700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7b00c8750 0x7ff7b00c8bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37482/0 (socket says 172.21.15.27:37482) 2023-10-06T01:42:08.718 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.713+0000 7ff7b4834700 1 -- 172.21.15.27:0/3344909149 learned_addr learned my addr 172.21.15.27:0/3344909149 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:08.718 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.713+0000 7ff7af7fe700 1 --2- 172.21.15.27:0/3344909149 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff7b015a4a0 0x7ff7b015a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:08.718 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.713+0000 7ff7affff700 1 --2- 172.21.15.27:0/3344909149 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff7b01596c0 0x7ff7b0157a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:08.718 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.713+0000 7ff7b4834700 1 -- 172.21.15.27:0/3344909149 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff7b01596c0 msgr2=0x7ff7b0157a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:08.718 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.713+0000 7ff7b4834700 1 --2- 172.21.15.27:0/3344909149 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff7b01596c0 0x7ff7b0157a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:08.718 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.713+0000 7ff7b4834700 1 -- 172.21.15.27:0/3344909149 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff7b015a4a0 msgr2=0x7ff7b015a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:08.719 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.713+0000 7ff7b4834700 1 --2- 172.21.15.27:0/3344909149 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff7b015a4a0 0x7ff7b015a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:08.719 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.713+0000 7ff7b4834700 1 -- 172.21.15.27:0/3344909149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7a40094f0 con 0x7ff7b00c8750 2023-10-06T01:42:08.719 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.713+0000 7ff7b4834700 1 --2- 172.21.15.27:0/3344909149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7b00c8750 0x7ff7b00c8bc0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff7a800ccc0 tx=0x7ff7a8001a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:08.719 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.713+0000 7ff797fff700 1 -- 172.21.15.27:0/3344909149 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a8005df0 con 0x7ff7b00c8750 2023-10-06T01:42:08.719 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.713+0000 7ff797fff700 1 -- 172.21.15.27:0/3344909149 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7a800a670 con 0x7ff7b00c8750 2023-10-06T01:42:08.719 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.713+0000 7ff7b6297700 1 -- 172.21.15.27:0/3344909149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7b015ae50 con 0x7ff7b00c8750 2023-10-06T01:42:08.719 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.713+0000 7ff797fff700 1 -- 172.21.15.27:0/3344909149 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7a800aa30 con 0x7ff7b00c8750 2023-10-06T01:42:08.719 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.713+0000 7ff7b6297700 1 -- 172.21.15.27:0/3344909149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7b015b0d0 con 0x7ff7b00c8750 2023-10-06T01:42:08.719 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.714+0000 7ff7b6297700 1 -- 172.21.15.27:0/3344909149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7b00c5aa0 con 0x7ff7b00c8750 2023-10-06T01:42:08.719 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.715+0000 7ff797fff700 1 -- 172.21.15.27:0/3344909149 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff7a80070f0 con 0x7ff7b00c8750 2023-10-06T01:42:08.719 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.715+0000 7ff797fff700 1 --2- 172.21.15.27:0/3344909149 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7ff79c066f30 0x7ff79c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:08.719 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.715+0000 7ff797fff700 1 -- 172.21.15.27:0/3344909149 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6058+0+0 (secure 0 0 0) 0x7ff7a80846c0 con 0x7ff7b00c8750 2023-10-06T01:42:08.719 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.716+0000 7ff7affff700 1 --2- 172.21.15.27:0/3344909149 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7ff79c066f30 0x7ff79c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:08.720 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.716+0000 7ff7affff700 1 --2- 172.21.15.27:0/3344909149 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7ff79c066f30 0x7ff79c0693e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff7a0009db0 tx=0x7ff7a00091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:08.721 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.720+0000 7ff797fff700 1 -- 172.21.15.27:0/3344909149 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff7a80517d0 con 0x7ff7b00c8750 2023-10-06T01:42:08.959 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.958+0000 7ff7b6297700 1 -- 172.21.15.27:0/3344909149 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7b0064050 con 0x7ff7b00c8750 2023-10-06T01:42:08.960 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.959+0000 7ff797fff700 1 -- 172.21.15.27:0/3344909149 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4597 (secure 0 0 0) 0x7ff7a8054df0 con 0x7ff7b00c8750 2023-10-06T01:42:08.960 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:08.960 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4870,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6836/1043044736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":1043044736},{"type":"v1","addr":"172.21.15.124:6836","nonce":1043044736}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4962,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/1207987117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1207987117},{"type":"v1","addr":"172.21.15.27:6834","nonce":1207987117}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4965,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/943261840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":943261840},{"type":"v1","addr":"172.21.15.27:6835","nonce":943261840}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T01:42:03.036784+0000","modified":"2023-10-06T01:42:06.652163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4974},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4974":{"gid":4974,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.124:6837/325247112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6835","nonce":325247112},{"type":"v1","addr":"172.21.15.124:6837","nonce":325247112}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-10-06T01:42:08.962 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.960+0000 7ff7b6297700 1 -- 172.21.15.27:0/3344909149 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7ff79c066f30 msgr2=0x7ff79c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:08.962 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.960+0000 7ff7b6297700 1 --2- 172.21.15.27:0/3344909149 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7ff79c066f30 0x7ff79c0693e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff7a0009db0 tx=0x7ff7a00091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:08.962 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.960+0000 7ff7b6297700 1 -- 172.21.15.27:0/3344909149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7b00c8750 msgr2=0x7ff7b00c8bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:08.962 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.960+0000 7ff7b6297700 1 --2- 172.21.15.27:0/3344909149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7b00c8750 0x7ff7b00c8bc0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff7a800ccc0 tx=0x7ff7a8001a20 comp rx=0 tx=0).stop 2023-10-06T01:42:08.962 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.961+0000 7ff7b6297700 1 -- 172.21.15.27:0/3344909149 shutdown_connections 2023-10-06T01:42:08.963 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.961+0000 7ff7b6297700 1 --2- 172.21.15.27:0/3344909149 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7ff79c066f30 0x7ff79c0693e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:08.963 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.961+0000 7ff7b6297700 1 --2- 172.21.15.27:0/3344909149 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff7b01596c0 0x7ff7b0157a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:08.963 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.961+0000 7ff7b6297700 1 --2- 172.21.15.27:0/3344909149 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff7b00c8750 0x7ff7b00c8bc0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:08.963 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.961+0000 7ff7b6297700 1 --2- 172.21.15.27:0/3344909149 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff7b015a4a0 0x7ff7b015a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:08.963 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.961+0000 7ff7b6297700 1 -- 172.21.15.27:0/3344909149 >> 172.21.15.27:0/3344909149 conn(0x7ff7b0067630 msgr2=0x7ff7b006ca50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:08.963 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.961+0000 7ff7b6297700 1 -- 172.21.15.27:0/3344909149 shutdown_connections 2023-10-06T01:42:08.963 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:08.961+0000 7ff7b6297700 1 -- 172.21.15.27:0/3344909149 wait complete. 2023-10-06T01:42:08.963 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 15 2023-10-06T01:42:09.376 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-06T01:42:03.036784+0000', 'modified': '2023-10-06T01:42:06.652163+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4974}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4974': {'gid': 4974, 'name': 'd', 'rank': 0, 'incarnation': 12, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.124:6837/325247112', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.124:6835', 'nonce': 325247112}, {'type': 'v1', 'addr': '172.21.15.124:6837', 'nonce': 325247112}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138320759226367, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2023-10-06T01:42:09.376 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-06T01:42:09.377 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-06T01:42:09.498 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.496+0000 7f6bb1e2e700 1 Processor -- start 2023-10-06T01:42:09.498 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.497+0000 7f6bb1e2e700 1 -- start start 2023-10-06T01:42:09.500 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.497+0000 7f6bb1e2e700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6bac159500 0x7f6bac1598d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:09.500 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.497+0000 7f6bb1e2e700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f6bac0b3fd0 0x7f6bac15d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:09.500 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.497+0000 7f6bb1e2e700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bac15dd20 con 0x7f6bac15d8e0 2023-10-06T01:42:09.500 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.497+0000 7f6bb1e2e700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bac0b1110 con 0x7f6bac159500 2023-10-06T01:42:09.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.497+0000 7f6bb1e2e700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bac0b13b0 con 0x7f6bac0b3fd0 2023-10-06T01:42:09.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.497+0000 7f6bab7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6bac159500 0x7f6bac1598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:09.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.498+0000 7f6babfff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6bac15d8e0 0x7f6bac0cf6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52116/0 (socket says 172.21.15.27:52116) 2023-10-06T01:42:09.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.498+0000 7f6bab7fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6bac159500 0x7f6bac1598d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37496/0 (socket says 172.21.15.27:37496) 2023-10-06T01:42:09.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.498+0000 7f6babfff700 1 -- 172.21.15.27:0/3653543981 learned_addr learned my addr 172.21.15.27:0/3653543981 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:09.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.498+0000 7f6baaffd700 1 --2- 172.21.15.27:0/3653543981 >> v2:172.21.15.124:3300/0 conn(0x7f6bac0b3fd0 0x7f6bac15d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:09.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.498+0000 7f6bab7fe700 1 -- 172.21.15.27:0/3653543981 >> v2:172.21.15.124:3300/0 conn(0x7f6bac0b3fd0 msgr2=0x7f6bac15d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:09.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.498+0000 7f6bab7fe700 1 --2- 172.21.15.27:0/3653543981 >> v2:172.21.15.124:3300/0 conn(0x7f6bac0b3fd0 0x7f6bac15d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.498+0000 7f6bab7fe700 1 -- 172.21.15.27:0/3653543981 >> v1:172.21.15.27:6789/0 conn(0x7f6bac15d8e0 legacy=0x7f6bac0cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:09.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.498+0000 7f6baa7fc700 1 -- 172.21.15.27:0/3653543981 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4057720556 0 0) 0x7f6bac15dd20 con 0x7f6bac15d8e0 2023-10-06T01:42:09.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.498+0000 7f6bab7fe700 1 -- 172.21.15.27:0/3653543981 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bac0b1690 con 0x7f6bac159500 2023-10-06T01:42:09.501 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.498+0000 7f6bab7fe700 1 --2- 172.21.15.27:0/3653543981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bac159500 0x7f6bac1598d0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f6bac063430 tx=0x7f6b9c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20fec2986738ad14 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:09.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.498+0000 7f6baa7fc700 1 -- 172.21.15.27:0/3653543981 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c00b6b0 con 0x7f6bac159500 2023-10-06T01:42:09.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.498+0000 7f6baa7fc700 1 -- 172.21.15.27:0/3653543981 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b9c00b810 con 0x7f6bac159500 2023-10-06T01:42:09.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.498+0000 7f6baa7fc700 1 -- 172.21.15.27:0/3653543981 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c00bbb0 con 0x7f6bac159500 2023-10-06T01:42:09.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.499+0000 7f6bb1e2e700 1 -- 172.21.15.27:0/3653543981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bac159500 msgr2=0x7f6bac1598d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:09.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.499+0000 7f6bb1e2e700 1 --2- 172.21.15.27:0/3653543981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bac159500 0x7f6bac1598d0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f6bac063430 tx=0x7f6b9c0091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.499+0000 7f6bb1e2e700 1 -- 172.21.15.27:0/3653543981 shutdown_connections 2023-10-06T01:42:09.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.499+0000 7f6bb1e2e700 1 --2- 172.21.15.27:0/3653543981 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bac159500 0x7f6bac1598d0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.499+0000 7f6bb1e2e700 1 --2- 172.21.15.27:0/3653543981 >> v2:172.21.15.124:3300/0 conn(0x7f6bac0b3fd0 0x7f6bac15d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.499+0000 7f6bb1e2e700 1 -- 172.21.15.27:0/3653543981 >> 172.21.15.27:0/3653543981 conn(0x7f6bac067630 msgr2=0x7f6bac069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:09.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.499+0000 7f6bb1e2e700 1 -- 172.21.15.27:0/3653543981 shutdown_connections 2023-10-06T01:42:09.502 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.499+0000 7f6bb1e2e700 1 -- 172.21.15.27:0/3653543981 wait complete. 2023-10-06T01:42:09.504 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.500+0000 7f6bb1e2e700 1 Processor -- start 2023-10-06T01:42:09.504 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.500+0000 7f6bb1e2e700 1 -- start start 2023-10-06T01:42:09.505 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.501+0000 7f6bb1e2e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6bac0b3fd0 0x7f6bac15c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:09.505 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.501+0000 7f6bb1e2e700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6bac159500 0x7f6bac155ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:09.505 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.501+0000 7f6bb1e2e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bac15d8e0 0x7f6bac1410d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:09.505 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.501+0000 7f6bb1e2e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bac15a290 con 0x7f6bac15d8e0 2023-10-06T01:42:09.505 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.501+0000 7f6bb1e2e700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bac141610 con 0x7f6bac159500 2023-10-06T01:42:09.505 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.501+0000 7f6bb1e2e700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bac159980 con 0x7f6bac0b3fd0 2023-10-06T01:42:09.505 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.501+0000 7f6babfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bac15d8e0 0x7f6bac1410d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:09.505 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.501+0000 7f6babfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bac15d8e0 0x7f6bac1410d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37510/0 (socket says 172.21.15.27:37510) 2023-10-06T01:42:09.505 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.501+0000 7f6babfff700 1 -- 172.21.15.27:0/1873436341 learned_addr learned my addr 172.21.15.27:0/1873436341 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:09.505 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.501+0000 7f6bab7fe700 1 --2- 172.21.15.27:0/1873436341 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6bac0b3fd0 0x7f6bac15c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:09.505 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.501+0000 7f6baaffd700 1 --2- 172.21.15.27:0/1873436341 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6bac159500 0x7f6bac155ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:09.505 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.501+0000 7f6babfff700 1 -- 172.21.15.27:0/1873436341 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6bac0b3fd0 msgr2=0x7f6bac15c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:09.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.501+0000 7f6babfff700 1 --2- 172.21.15.27:0/1873436341 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6bac0b3fd0 0x7f6bac15c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.501+0000 7f6babfff700 1 -- 172.21.15.27:0/1873436341 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6bac159500 msgr2=0x7f6bac155ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:09.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.501+0000 7f6babfff700 1 --2- 172.21.15.27:0/1873436341 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6bac159500 0x7f6bac155ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.501+0000 7f6babfff700 1 -- 172.21.15.27:0/1873436341 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b9c009420 con 0x7f6bac15d8e0 2023-10-06T01:42:09.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.502+0000 7f6babfff700 1 --2- 172.21.15.27:0/1873436341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bac15d8e0 0x7f6bac1410d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6ba400cbf0 tx=0x7f6ba40019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:09.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.502+0000 7f6b937fe700 1 -- 172.21.15.27:0/1873436341 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba400a9c0 con 0x7f6bac15d8e0 2023-10-06T01:42:09.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.502+0000 7f6b937fe700 1 -- 172.21.15.27:0/1873436341 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba400ab20 con 0x7f6bac15d8e0 2023-10-06T01:42:09.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.502+0000 7f6b937fe700 1 -- 172.21.15.27:0/1873436341 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba40070f0 con 0x7f6bac15d8e0 2023-10-06T01:42:09.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.502+0000 7f6bb1e2e700 1 -- 172.21.15.27:0/1873436341 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bac15c980 con 0x7f6bac15d8e0 2023-10-06T01:42:09.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.502+0000 7f6bb1e2e700 1 -- 172.21.15.27:0/1873436341 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bac159f20 con 0x7f6bac15d8e0 2023-10-06T01:42:09.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.503+0000 7f6b937fe700 1 -- 172.21.15.27:0/1873436341 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6ba4007520 con 0x7f6bac15d8e0 2023-10-06T01:42:09.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.504+0000 7f6b937fe700 1 --2- 172.21.15.27:0/1873436341 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6b98066f30 0x7f6b980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:09.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.504+0000 7f6b937fe700 1 -- 172.21.15.27:0/1873436341 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f6ba4084a90 con 0x7f6bac15d8e0 2023-10-06T01:42:09.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.504+0000 7f6bab7fe700 1 --2- 172.21.15.27:0/1873436341 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6b98066f30 0x7f6b980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:09.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.505+0000 7f6bab7fe700 1 --2- 172.21.15.27:0/1873436341 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6b98066f30 0x7f6b980693e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c0093f0 tx=0x7f6b9c017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:09.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.506+0000 7f6bbb126b80 1 Processor -- start 2023-10-06T01:42:09.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.506+0000 7f6bbb126b80 1 -- start start 2023-10-06T01:42:09.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.507+0000 7f6bbb126b80 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x558b93dfa840 0x558b93dfac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:09.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.507+0000 7f6bbb126b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b93dfbcc0 0x558b93e4a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:09.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.507+0000 7f6bbb126b80 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x558b93e4ac00 0x558b93e4b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:09.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.507+0000 7f6bbb126b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b93d51a70 con 0x558b93dfbcc0 2023-10-06T01:42:09.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.507+0000 7f6bbb126b80 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b93d4f270 con 0x558b93dfa840 2023-10-06T01:42:09.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.507+0000 7f6bbb126b80 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x558b93d4b060 con 0x558b93e4ac00 2023-10-06T01:42:09.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.507+0000 7f6baaffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b93dfbcc0 0x558b93e4a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:09.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.507+0000 7f6baaffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b93dfbcc0 0x558b93e4a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37512/0 (socket says 172.21.15.27:37512) 2023-10-06T01:42:09.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.507+0000 7f6baaffd700 1 -- 172.21.15.27:0/1101703066 learned_addr learned my addr 172.21.15.27:0/1101703066 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:09.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.507+0000 7f6bab7fe700 1 --2- 172.21.15.27:0/1101703066 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x558b93dfa840 0x558b93dfac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:09.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.507+0000 7f6babfff700 1 --2- 172.21.15.27:0/1101703066 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x558b93e4ac00 0x558b93e4b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:09.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.507+0000 7f6baaffd700 1 -- 172.21.15.27:0/1101703066 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x558b93e4ac00 msgr2=0x558b93e4b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:09.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.507+0000 7f6baaffd700 1 --2- 172.21.15.27:0/1101703066 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x558b93e4ac00 0x558b93e4b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.507+0000 7f6baaffd700 1 -- 172.21.15.27:0/1101703066 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x558b93dfa840 msgr2=0x558b93dfac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:09.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.507+0000 7f6baaffd700 1 --2- 172.21.15.27:0/1101703066 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x558b93dfa840 0x558b93dfac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.507+0000 7f6baaffd700 1 -- 172.21.15.27:0/1101703066 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558b93be5270 con 0x558b93dfbcc0 2023-10-06T01:42:09.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.507+0000 7f6baaffd700 1 --2- 172.21.15.27:0/1101703066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b93dfbcc0 0x558b93e4a6c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6ba00079d0 tx=0x7f6ba0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca8280ea51b461da server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:09.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.508+0000 7f6b90ff9700 1 -- 172.21.15.27:0/1101703066 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba000f670 con 0x558b93dfbcc0 2023-10-06T01:42:09.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.508+0000 7f6b90ff9700 1 -- 172.21.15.27:0/1101703066 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba000f7d0 con 0x558b93dfbcc0 2023-10-06T01:42:09.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.508+0000 7f6b90ff9700 1 -- 172.21.15.27:0/1101703066 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba000fb70 con 0x558b93dfbcc0 2023-10-06T01:42:09.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.508+0000 7f6bbb126b80 1 -- 172.21.15.27:0/1101703066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b93dfbcc0 msgr2=0x558b93e4a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:09.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.508+0000 7f6bbb126b80 1 --2- 172.21.15.27:0/1101703066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b93dfbcc0 0x558b93e4a6c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6ba00079d0 tx=0x7f6ba0008040 comp rx=0 tx=0).stop 2023-10-06T01:42:09.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.508+0000 7f6bbb126b80 1 -- 172.21.15.27:0/1101703066 shutdown_connections 2023-10-06T01:42:09.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.508+0000 7f6bbb126b80 1 --2- 172.21.15.27:0/1101703066 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x558b93e4ac00 0x558b93e4b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.508+0000 7f6bbb126b80 1 --2- 172.21.15.27:0/1101703066 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x558b93dfa840 0x558b93dfac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.508+0000 7f6bbb126b80 1 --2- 172.21.15.27:0/1101703066 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b93dfbcc0 0x558b93e4a6c0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.508+0000 7f6bbb126b80 1 -- 172.21.15.27:0/1101703066 >> 172.21.15.27:0/1101703066 conn(0x558b93dfc160 msgr2=0x558b93dfc530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:09.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.508+0000 7f6bbb126b80 1 -- 172.21.15.27:0/1101703066 shutdown_connections 2023-10-06T01:42:09.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.508+0000 7f6bbb126b80 1 -- 172.21.15.27:0/1101703066 wait complete. 2023-10-06T01:42:09.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.509+0000 7f6bbb126b80 1 -- start start 2023-10-06T01:42:09.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.509+0000 7f6bbb126b80 1 Processor -- start 2023-10-06T01:42:09.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.510+0000 7f6bbb126b80 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x558b93dfa840 0x558b93e298d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:09.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.510+0000 7f6bbb126b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b93dfbcc0 0x558b93e29e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:09.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.510+0000 7f6bbb126b80 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x558b93e4ac00 0x558b93e24fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:09.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.510+0000 7f6bbb126b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b93cf6890 con 0x558b93dfbcc0 2023-10-06T01:42:09.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.510+0000 7f6bbb126b80 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b93d11ea0 con 0x558b93dfa840 2023-10-06T01:42:09.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.510+0000 7f6bbb126b80 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x558b93bcde30 con 0x558b93e4ac00 2023-10-06T01:42:09.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.510+0000 7f6bbb126b80 10 client.0 _ll_register_callbacks cb 0x558b93dbde70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-06T01:42:09.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.510+0000 7f6baaffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b93dfbcc0 0x558b93e29e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:09.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.510+0000 7f6baaffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b93dfbcc0 0x558b93e29e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37518/0 (socket says 172.21.15.27:37518) 2023-10-06T01:42:09.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.510+0000 7f6baaffd700 1 -- 172.21.15.27:0/3515377724 learned_addr learned my addr 172.21.15.27:0/3515377724 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:09.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.510+0000 7f6bab7fe700 1 --2- 172.21.15.27:0/3515377724 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x558b93dfa840 0x558b93e298d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:09.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.510+0000 7f6babfff700 1 --2- 172.21.15.27:0/3515377724 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x558b93e4ac00 0x558b93e24fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:09.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.510+0000 7f6baaffd700 1 -- 172.21.15.27:0/3515377724 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x558b93e4ac00 msgr2=0x558b93e24fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:09.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.510+0000 7f6baaffd700 1 --2- 172.21.15.27:0/3515377724 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x558b93e4ac00 0x558b93e24fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.510+0000 7f6baaffd700 1 -- 172.21.15.27:0/3515377724 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x558b93dfa840 msgr2=0x558b93e298d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:09.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.510+0000 7f6baaffd700 1 --2- 172.21.15.27:0/3515377724 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x558b93dfa840 0x558b93e298d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.510+0000 7f6baaffd700 1 -- 172.21.15.27:0/3515377724 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ba000e040 con 0x558b93dfbcc0 2023-10-06T01:42:09.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.510+0000 7f6baaffd700 1 --2- 172.21.15.27:0/3515377724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b93dfbcc0 0x558b93e29e10 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0016000 tx=0x7f6ba001c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed496006235c9944 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:09.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.510+0000 7f6b89ffb700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2023-10-06T01:42:09.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.511+0000 7f6b89ffb700 1 -- 172.21.15.27:0/3515377724 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba0023070 con 0x558b93dfbcc0 2023-10-06T01:42:09.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.511+0000 7f6b89ffb700 1 -- 172.21.15.27:0/3515377724 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(31..31 src has 1..31) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f6ba000f5f0 con 0x558b93dfbcc0 2023-10-06T01:42:09.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.511+0000 7f6bbb126b80 1 -- 172.21.15.27:0/3515377724 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558b93def940 con 0x558b93dfbcc0 2023-10-06T01:42:09.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.511+0000 7f6b89ffb700 1 -- 172.21.15.27:0/3515377724 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ba000fee0 con 0x558b93dfbcc0 2023-10-06T01:42:09.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.511+0000 7f6bbb126b80 10 client.5154 fetch_fsmap learned FSMap version 15 2023-10-06T01:42:09.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.511+0000 7f6bbb126b80 1 -- 172.21.15.27:0/3515377724 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) v3 -- 0x558b93b87fe0 con 0x558b93dfbcc0 2023-10-06T01:42:09.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.511+0000 7f6b89ffb700 1 -- 172.21.15.27:0/3515377724 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558b93def940 con 0x558b93dfbcc0 2023-10-06T01:42:09.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.511+0000 7f6b89ffb700 1 -- 172.21.15.27:0/3515377724 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 15) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f6ba001d420 con 0x558b93dfbcc0 2023-10-06T01:42:09.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.511+0000 7f6bbb126b80 10 client.5154 fetch_fsmap finished waiting for FSMap version 15 2023-10-06T01:42:09.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.511+0000 7f6bbb126b80 10 client.5154 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-06T01:42:09.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.511+0000 7f6bbb126b80 20 client.5154 populate_metadata read hostname 'smithi027' 2023-10-06T01:42:09.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.511+0000 7f6bbb126b80 1 --2- 172.21.15.27:0/3515377724 >> [v2:172.21.15.124:6835/325247112,v1:172.21.15.124:6837/325247112] conn(0x558b93e548f0 0x558b93e54cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:09.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.511+0000 7f6bbb126b80 4 client.5154 mds_command: new command op to 4974 tid=0[{"prefix": "get_command_descriptions"}] 2023-10-06T01:42:09.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.511+0000 7f6bbb126b80 1 -- 172.21.15.27:0/3515377724 --> [v2:172.21.15.124:6835/325247112,v1:172.21.15.124:6837/325247112] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558b93dcc870 con 0x558b93e548f0 2023-10-06T01:42:09.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.511+0000 7f6bab7fe700 1 --2- 172.21.15.27:0/3515377724 >> [v2:172.21.15.124:6835/325247112,v1:172.21.15.124:6837/325247112] conn(0x558b93e548f0 0x558b93e54cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:09.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.512+0000 7f6b89ffb700 10 client.5154 ms_handle_connect on v2:172.21.15.124:6835/325247112 2023-10-06T01:42:09.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.512+0000 7f6bab7fe700 1 --2- 172.21.15.27:0/3515377724 >> [v2:172.21.15.124:6835/325247112,v1:172.21.15.124:6837/325247112] conn(0x558b93e548f0 0x558b93e54cc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b8dd06d68a79c40 server_cookie=93cba5c9eeb6ad89 in_seq=0 out_seq=0 2023-10-06T01:42:09.520 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.512+0000 7f6b89ffb700 1 -- 172.21.15.27:0/3515377724 <== mds.0 v2:172.21.15.124:6835/325247112 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+16954 (crc 0 0 0) 0x558b93dcc870 con 0x558b93e548f0 2023-10-06T01:42:09.520 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.512+0000 7f6b89ffb700 10 client.5154 handle_command_reply: tid=0 2023-10-06T01:42:09.520 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.512+0000 7f6bbb126b80 1 client.5154 shutdown 2023-10-06T01:42:09.520 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.513+0000 7f6bbb126b80 1 -- 172.21.15.27:0/3515377724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b93dfbcc0 msgr2=0x558b93e29e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:09.520 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.513+0000 7f6bbb126b80 1 --2- 172.21.15.27:0/3515377724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b93dfbcc0 0x558b93e29e10 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6ba0016000 tx=0x7f6ba001c000 comp rx=0 tx=0).stop 2023-10-06T01:42:09.520 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.513+0000 7f6bbb126b80 1 -- 172.21.15.27:0/3515377724 shutdown_connections 2023-10-06T01:42:09.520 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.513+0000 7f6bbb126b80 1 --2- 172.21.15.27:0/3515377724 >> [v2:172.21.15.124:6835/325247112,v1:172.21.15.124:6837/325247112] conn(0x558b93e548f0 0x558b93e54cc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.520 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.513+0000 7f6bbb126b80 1 --2- 172.21.15.27:0/3515377724 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x558b93e4ac00 0x558b93e24fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.520 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.513+0000 7f6bbb126b80 1 --2- 172.21.15.27:0/3515377724 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x558b93dfa840 0x558b93e298d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.520 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.513+0000 7f6b89ffb700 0 client.5154 ms_handle_reset on v2:172.21.15.124:6835/325247112 2023-10-06T01:42:09.520 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.513+0000 7f6bbb126b80 1 --2- 172.21.15.27:0/3515377724 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b93dfbcc0 0x558b93e29e10 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.521 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.513+0000 7f6bbb126b80 1 -- 172.21.15.27:0/3515377724 >> 172.21.15.27:0/3515377724 conn(0x558b93dfc160 msgr2=0x558b93dfb1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:09.521 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.513+0000 7f6bbb126b80 1 -- 172.21.15.27:0/3515377724 shutdown_connections 2023-10-06T01:42:09.521 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.513+0000 7f6bbb126b80 1 -- 172.21.15.27:0/3515377724 wait complete. 2023-10-06T01:42:09.521 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.513+0000 7f6bbb126b80 20 client.5154 trim_cache size 0 max 16384 2023-10-06T01:42:09.531 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.530+0000 7f6bbb126b80 1 Processor -- start 2023-10-06T01:42:09.531 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.530+0000 7f6bbb126b80 1 -- start start 2023-10-06T01:42:09.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.531+0000 7f6bbb126b80 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x558b93dfa840 0x558b93e23ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:09.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.531+0000 7f6bbb126b80 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x558b93dfbcc0 0x558b93e243e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:09.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.531+0000 7f6bbb126b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b93e4ac00 0x558b93e5acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:09.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.531+0000 7f6bbb126b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b93e3e330 con 0x558b93e4ac00 2023-10-06T01:42:09.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.531+0000 7f6bbb126b80 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b93e23a90 con 0x558b93dfbcc0 2023-10-06T01:42:09.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.531+0000 7f6bbb126b80 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x558b93df8f70 con 0x558b93dfa840 2023-10-06T01:42:09.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.531+0000 7f6babfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b93e4ac00 0x558b93e5acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:09.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.531+0000 7f6babfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b93e4ac00 0x558b93e5acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37522/0 (socket says 172.21.15.27:37522) 2023-10-06T01:42:09.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.531+0000 7f6babfff700 1 -- 172.21.15.27:0/3153803188 learned_addr learned my addr 172.21.15.27:0/3153803188 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:09.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.531+0000 7f6bab7fe700 1 --2- 172.21.15.27:0/3153803188 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x558b93dfa840 0x558b93e23ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:09.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.531+0000 7f6baaffd700 1 --2- 172.21.15.27:0/3153803188 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x558b93dfbcc0 0x558b93e243e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:09.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.531+0000 7f6babfff700 1 -- 172.21.15.27:0/3153803188 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x558b93dfa840 msgr2=0x558b93e23ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:09.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.531+0000 7f6babfff700 1 --2- 172.21.15.27:0/3153803188 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x558b93dfa840 0x558b93e23ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.531+0000 7f6babfff700 1 -- 172.21.15.27:0/3153803188 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x558b93dfbcc0 msgr2=0x558b93e243e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:09.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.531+0000 7f6babfff700 1 --2- 172.21.15.27:0/3153803188 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x558b93dfbcc0 0x558b93e243e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.531+0000 7f6babfff700 1 -- 172.21.15.27:0/3153803188 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558b93d19780 con 0x558b93e4ac00 2023-10-06T01:42:09.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.531+0000 7f6baaffd700 1 --2- 172.21.15.27:0/3153803188 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x558b93dfbcc0 0x558b93e243e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:42:09.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.531+0000 7f6babfff700 1 --2- 172.21.15.27:0/3153803188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b93e4ac00 0x558b93e5acf0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f6ba400afd0 tx=0x7f6ba4082d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49d2262660460cf3 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:09.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.531+0000 7f6b917fa700 1 -- 172.21.15.27:0/3153803188 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba4058b10 con 0x558b93e4ac00 2023-10-06T01:42:09.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.531+0000 7f6b917fa700 1 -- 172.21.15.27:0/3153803188 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba405c860 con 0x558b93e4ac00 2023-10-06T01:42:09.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.531+0000 7f6b917fa700 1 -- 172.21.15.27:0/3153803188 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba4053c60 con 0x558b93e4ac00 2023-10-06T01:42:09.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.532+0000 7f6bbb126b80 1 -- 172.21.15.27:0/3153803188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b93e4ac00 msgr2=0x558b93e5acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:09.536 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.532+0000 7f6bbb126b80 1 --2- 172.21.15.27:0/3153803188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b93e4ac00 0x558b93e5acf0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f6ba400afd0 tx=0x7f6ba4082d90 comp rx=0 tx=0).stop 2023-10-06T01:42:09.536 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.532+0000 7f6bbb126b80 1 -- 172.21.15.27:0/3153803188 shutdown_connections 2023-10-06T01:42:09.536 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.532+0000 7f6bbb126b80 1 --2- 172.21.15.27:0/3153803188 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x558b93dfa840 0x558b93e23ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.536 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.532+0000 7f6bbb126b80 1 --2- 172.21.15.27:0/3153803188 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b93e4ac00 0x558b93e5acf0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.536 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.532+0000 7f6bbb126b80 1 --2- 172.21.15.27:0/3153803188 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x558b93dfbcc0 0x558b93e243e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.536 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.532+0000 7f6bbb126b80 1 -- 172.21.15.27:0/3153803188 >> 172.21.15.27:0/3153803188 conn(0x558b93dfc160 msgr2=0x558b93df8860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:09.536 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.532+0000 7f6bbb126b80 1 -- 172.21.15.27:0/3153803188 shutdown_connections 2023-10-06T01:42:09.536 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.532+0000 7f6bbb126b80 1 -- 172.21.15.27:0/3153803188 wait complete. 2023-10-06T01:42:09.536 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.532+0000 7f6bbb126b80 1 -- start start 2023-10-06T01:42:09.536 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.532+0000 7f6bbb126b80 1 Processor -- start 2023-10-06T01:42:09.540 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.533+0000 7f6bbb126b80 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x558b93dfa840 0x558b93e23ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:09.540 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.533+0000 7f6bbb126b80 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x558b93dfbcc0 0x558b93e243e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:09.540 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.533+0000 7f6bbb126b80 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b93e4ac00 0x558b93e21a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:09.540 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.533+0000 7f6bbb126b80 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b93e232c0 con 0x558b93e4ac00 2023-10-06T01:42:09.540 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.533+0000 7f6bbb126b80 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b93ea7ec0 con 0x558b93dfbcc0 2023-10-06T01:42:09.540 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.533+0000 7f6bbb126b80 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x558b93e45280 con 0x558b93dfa840 2023-10-06T01:42:09.540 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.533+0000 7f6bbb126b80 10 client.0 _ll_register_callbacks cb 0x558b93dbde70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-06T01:42:09.540 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.533+0000 7f6babfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b93e4ac00 0x558b93e21a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:09.541 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.533+0000 7f6babfff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b93e4ac00 0x558b93e21a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37528/0 (socket says 172.21.15.27:37528) 2023-10-06T01:42:09.541 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.533+0000 7f6babfff700 1 -- 172.21.15.27:0/3595710369 learned_addr learned my addr 172.21.15.27:0/3595710369 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:09.541 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.533+0000 7f6bab7fe700 1 --2- 172.21.15.27:0/3595710369 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x558b93dfa840 0x558b93e23ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:09.541 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.533+0000 7f6baaffd700 1 --2- 172.21.15.27:0/3595710369 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x558b93dfbcc0 0x558b93e243e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:09.541 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.534+0000 7f6babfff700 1 -- 172.21.15.27:0/3595710369 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x558b93dfa840 msgr2=0x558b93e23ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:09.541 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.534+0000 7f6babfff700 1 --2- 172.21.15.27:0/3595710369 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x558b93dfa840 0x558b93e23ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.541 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.534+0000 7f6babfff700 1 -- 172.21.15.27:0/3595710369 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x558b93dfbcc0 msgr2=0x558b93e243e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:09.541 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.534+0000 7f6babfff700 1 --2- 172.21.15.27:0/3595710369 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x558b93dfbcc0 0x558b93e243e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.541 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.534+0000 7f6babfff700 1 -- 172.21.15.27:0/3595710369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bac159f20 con 0x558b93e4ac00 2023-10-06T01:42:09.541 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.534+0000 7f6bab7fe700 1 --2- 172.21.15.27:0/3595710369 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x558b93dfa840 0x558b93e23ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:42:09.541 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.534+0000 7f6baaffd700 1 --2- 172.21.15.27:0/3595710369 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x558b93dfbcc0 0x558b93e243e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:42:09.541 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.534+0000 7f6babfff700 1 --2- 172.21.15.27:0/3595710369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b93e4ac00 0x558b93e21a20 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6ba400afd0 tx=0x7f6ba4082000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3c938fd90578b18 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:09.542 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.534+0000 7f6b89ffb700 10 client.0 ms_handle_connect on v2:172.21.15.27:3300/0 2023-10-06T01:42:09.542 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.534+0000 7f6b89ffb700 1 -- 172.21.15.27:0/3595710369 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba4061070 con 0x558b93e4ac00 2023-10-06T01:42:09.542 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.534+0000 7f6b89ffb700 1 -- 172.21.15.27:0/3595710369 <== mon.0 v2:172.21.15.27:3300/0 2 ==== osd_map(31..31 src has 1..31) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f6ba4053c60 con 0x558b93e4ac00 2023-10-06T01:42:09.542 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.534+0000 7f6bbb126b80 1 -- 172.21.15.27:0/3595710369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6ba000f7d0 con 0x558b93e4ac00 2023-10-06T01:42:09.542 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.534+0000 7f6b89ffb700 1 -- 172.21.15.27:0/3595710369 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ba405c5e0 con 0x558b93e4ac00 2023-10-06T01:42:09.542 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.534+0000 7f6bbb126b80 10 client.5160 fetch_fsmap learned FSMap version 15 2023-10-06T01:42:09.542 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.534+0000 7f6bbb126b80 1 -- 172.21.15.27:0/3595710369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) v3 -- 0x558b93d00470 con 0x558b93e4ac00 2023-10-06T01:42:09.542 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.534+0000 7f6b89ffb700 1 -- 172.21.15.27:0/3595710369 <== mon.0 v2:172.21.15.27:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba000f7d0 con 0x558b93e4ac00 2023-10-06T01:42:09.542 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.534+0000 7f6b89ffb700 1 -- 172.21.15.27:0/3595710369 <== mon.0 v2:172.21.15.27:3300/0 5 ==== fsmap(e 15) v1 ==== 3100+0+0 (secure 0 0 0) 0x7f6ba4064070 con 0x558b93e4ac00 2023-10-06T01:42:09.542 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.534+0000 7f6bbb126b80 10 client.5160 fetch_fsmap finished waiting for FSMap version 15 2023-10-06T01:42:09.542 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.534+0000 7f6bbb126b80 10 client.5160 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-06T01:42:09.543 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.534+0000 7f6bbb126b80 20 client.5160 populate_metadata read hostname 'smithi027' 2023-10-06T01:42:09.543 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.534+0000 7f6bbb126b80 1 --2- 172.21.15.27:0/3595710369 >> [v2:172.21.15.124:6835/325247112,v1:172.21.15.124:6837/325247112] conn(0x558b93e221c0 0x558b93ea6690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:09.543 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.534+0000 7f6bbb126b80 4 client.5160 mds_command: new command op to 4974 tid=0[{"prefix": "status"}] 2023-10-06T01:42:09.543 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.534+0000 7f6bbb126b80 1 -- 172.21.15.27:0/3595710369 --> [v2:172.21.15.124:6835/325247112,v1:172.21.15.124:6837/325247112] -- command(tid 0: {"prefix": "status"}) v1 -- 0x558b93cede00 con 0x558b93e221c0 2023-10-06T01:42:09.543 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.535+0000 7f6bab7fe700 1 --2- 172.21.15.27:0/3595710369 >> [v2:172.21.15.124:6835/325247112,v1:172.21.15.124:6837/325247112] conn(0x558b93e221c0 0x558b93ea6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:09.543 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.535+0000 7f6b89ffb700 10 client.5160 ms_handle_connect on v2:172.21.15.124:6835/325247112 2023-10-06T01:42:09.543 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.535+0000 7f6bab7fe700 1 --2- 172.21.15.27:0/3595710369 >> [v2:172.21.15.124:6835/325247112,v1:172.21.15.124:6837/325247112] conn(0x558b93e221c0 0x558b93ea6690 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c0bdf9d2edc9a73 server_cookie=2098ee46ae23989d in_seq=0 out_seq=0 2023-10-06T01:42:09.543 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.535+0000 7f6b89ffb700 1 -- 172.21.15.27:0/3595710369 <== mds.0 v2:172.21.15.124:6835/325247112 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x558b93cede00 con 0x558b93e221c0 2023-10-06T01:42:09.543 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.535+0000 7f6b89ffb700 10 client.5160 handle_command_reply: tid=0 2023-10-06T01:42:09.543 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.535+0000 7f6bbb126b80 1 client.5160 shutdown 2023-10-06T01:42:09.543 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.535+0000 7f6bbb126b80 1 -- 172.21.15.27:0/3595710369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b93e4ac00 msgr2=0x558b93e21a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:09.544 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.535+0000 7f6bbb126b80 1 --2- 172.21.15.27:0/3595710369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b93e4ac00 0x558b93e21a20 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f6ba400afd0 tx=0x7f6ba4082000 comp rx=0 tx=0).stop 2023-10-06T01:42:09.544 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.535+0000 7f6bbb126b80 1 -- 172.21.15.27:0/3595710369 shutdown_connections 2023-10-06T01:42:09.544 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.535+0000 7f6bbb126b80 1 --2- 172.21.15.27:0/3595710369 >> [v2:172.21.15.124:6835/325247112,v1:172.21.15.124:6837/325247112] conn(0x558b93e221c0 0x558b93ea6690 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.544 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.535+0000 7f6bbb126b80 1 --2- 172.21.15.27:0/3595710369 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x558b93dfa840 0x558b93e23ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.544 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.535+0000 7f6bbb126b80 1 --2- 172.21.15.27:0/3595710369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x558b93e4ac00 0x558b93e21a20 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.544 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.535+0000 7f6b89ffb700 0 client.5160 ms_handle_reset on v2:172.21.15.124:6835/325247112 2023-10-06T01:42:09.544 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.535+0000 7f6bbb126b80 1 --2- 172.21.15.27:0/3595710369 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x558b93dfbcc0 0x558b93e243e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.544 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.535+0000 7f6bbb126b80 1 -- 172.21.15.27:0/3595710369 >> 172.21.15.27:0/3595710369 conn(0x558b93dfc160 msgr2=0x558b93df8860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:09.544 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.535+0000 7f6bbb126b80 1 -- 172.21.15.27:0/3595710369 shutdown_connections 2023-10-06T01:42:09.544 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.535+0000 7f6bbb126b80 1 -- 172.21.15.27:0/3595710369 wait complete. 2023-10-06T01:42:09.544 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.535+0000 7f6bbb126b80 20 client.5160 trim_cache size 0 max 16384 2023-10-06T01:42:09.545 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2023-10-06T01:42:09.545 INFO:teuthology.orchestra.run.smithi027.stdout: "cluster_fsid": "20a7bff0-74f8-4f2f-b6c6-899e804bc965", 2023-10-06T01:42:09.545 INFO:teuthology.orchestra.run.smithi027.stdout: "whoami": 0, 2023-10-06T01:42:09.545 INFO:teuthology.orchestra.run.smithi027.stdout: "id": 4974, 2023-10-06T01:42:09.545 INFO:teuthology.orchestra.run.smithi027.stdout: "want_state": "up:active", 2023-10-06T01:42:09.545 INFO:teuthology.orchestra.run.smithi027.stdout: "state": "up:active", 2023-10-06T01:42:09.545 INFO:teuthology.orchestra.run.smithi027.stdout: "fs_name": "cephfs", 2023-10-06T01:42:09.545 INFO:teuthology.orchestra.run.smithi027.stdout: "rank_uptime": 5.9044450489999996, 2023-10-06T01:42:09.545 INFO:teuthology.orchestra.run.smithi027.stdout: "mdsmap_epoch": 15, 2023-10-06T01:42:09.545 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch": 31, 2023-10-06T01:42:09.545 INFO:teuthology.orchestra.run.smithi027.stdout: "osdmap_epoch_barrier": 31, 2023-10-06T01:42:09.546 INFO:teuthology.orchestra.run.smithi027.stdout: "uptime": 13.773676744999999 2023-10-06T01:42:09.546 INFO:teuthology.orchestra.run.smithi027.stdout:} 2023-10-06T01:42:09.546 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.536+0000 7f6bb1e2e700 1 -- 172.21.15.27:0/1873436341 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6b98066f30 msgr2=0x7f6b980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:09.547 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.536+0000 7f6bb1e2e700 1 --2- 172.21.15.27:0/1873436341 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6b98066f30 0x7f6b980693e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c0093f0 tx=0x7f6b9c017000 comp rx=0 tx=0).stop 2023-10-06T01:42:09.547 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.536+0000 7f6bb1e2e700 1 -- 172.21.15.27:0/1873436341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bac15d8e0 msgr2=0x7f6bac1410d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:09.547 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.536+0000 7f6bb1e2e700 1 --2- 172.21.15.27:0/1873436341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bac15d8e0 0x7f6bac1410d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f6ba400cbf0 tx=0x7f6ba40019b0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.547 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.536+0000 7f6bb1e2e700 1 -- 172.21.15.27:0/1873436341 shutdown_connections 2023-10-06T01:42:09.547 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.536+0000 7f6bb1e2e700 1 --2- 172.21.15.27:0/1873436341 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6b98066f30 0x7f6b980693e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.547 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.536+0000 7f6bb1e2e700 1 --2- 172.21.15.27:0/1873436341 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6bac0b3fd0 0x7f6bac15c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.547 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.536+0000 7f6bb1e2e700 1 --2- 172.21.15.27:0/1873436341 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bac15d8e0 0x7f6bac1410d0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.547 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.536+0000 7f6bb1e2e700 1 --2- 172.21.15.27:0/1873436341 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6bac159500 0x7f6bac155ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.547 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.536+0000 7f6bb1e2e700 1 -- 172.21.15.27:0/1873436341 >> 172.21.15.27:0/1873436341 conn(0x7f6bac067630 msgr2=0x7f6bac14d390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:09.547 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.536+0000 7f6bb1e2e700 1 -- 172.21.15.27:0/1873436341 shutdown_connections 2023-10-06T01:42:09.547 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.536+0000 7f6bb1e2e700 1 -- 172.21.15.27:0/1873436341 wait complete. 2023-10-06T01:42:09.549 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-06T01:42:09.667 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.666+0000 7ff120584700 1 Processor -- start 2023-10-06T01:42:09.667 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.666+0000 7ff120584700 1 -- start start 2023-10-06T01:42:09.670 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.667+0000 7ff120584700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff1180cdc00 0x7ff1180cbe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:09.670 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.667+0000 7ff120584700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7ff1180c6dc0 0x7ff1180ae5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:09.670 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.667+0000 7ff120584700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1180ae140 con 0x7ff1180d1030 2023-10-06T01:42:09.670 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.667+0000 7ff120584700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1180cc390 con 0x7ff1180cdc00 2023-10-06T01:42:09.670 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.667+0000 7ff120584700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1180ad530 con 0x7ff1180c6dc0 2023-10-06T01:42:09.670 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.667+0000 7ff11e320700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff1180cdc00 0x7ff1180cbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:09.670 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.667+0000 7ff11db1f700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff1180d1030 0x7ff1180c66b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52130/0 (socket says 172.21.15.27:52130) 2023-10-06T01:42:09.670 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.667+0000 7ff11db1f700 1 -- 172.21.15.27:0/1556930637 learned_addr learned my addr 172.21.15.27:0/1556930637 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:09.671 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.667+0000 7ff11eb21700 1 --2- 172.21.15.27:0/1556930637 >> v2:172.21.15.124:3300/0 conn(0x7ff1180c6dc0 0x7ff1180ae5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:09.671 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.667+0000 7ff11e320700 1 -- 172.21.15.27:0/1556930637 >> v2:172.21.15.124:3300/0 conn(0x7ff1180c6dc0 msgr2=0x7ff1180ae5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:09.671 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.667+0000 7ff11e320700 1 --2- 172.21.15.27:0/1556930637 >> v2:172.21.15.124:3300/0 conn(0x7ff1180c6dc0 0x7ff1180ae5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.671 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.668+0000 7ff11e320700 1 -- 172.21.15.27:0/1556930637 >> v1:172.21.15.27:6789/0 conn(0x7ff1180d1030 legacy=0x7ff1180c66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:09.671 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.668+0000 7ff11e320700 1 -- 172.21.15.27:0/1556930637 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1180ad810 con 0x7ff1180cdc00 2023-10-06T01:42:09.671 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.668+0000 7ff11d31e700 1 -- 172.21.15.27:0/1556930637 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4273697272 0 0) 0x7ff1180ae140 con 0x7ff1180d1030 2023-10-06T01:42:09.671 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.668+0000 7ff11e320700 1 --2- 172.21.15.27:0/1556930637 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1180cdc00 0x7ff1180cbe50 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff11808a900 tx=0x7ff108009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11ad1088ab808be server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:09.671 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.668+0000 7ff11d31e700 1 -- 172.21.15.27:0/1556930637 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10800e070 con 0x7ff1180cdc00 2023-10-06T01:42:09.671 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.668+0000 7ff11d31e700 1 -- 172.21.15.27:0/1556930637 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff10800bb60 con 0x7ff1180cdc00 2023-10-06T01:42:09.671 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.668+0000 7ff11d31e700 1 -- 172.21.15.27:0/1556930637 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff108012430 con 0x7ff1180cdc00 2023-10-06T01:42:09.671 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.668+0000 7ff120584700 1 -- 172.21.15.27:0/1556930637 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1180cdc00 msgr2=0x7ff1180cbe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:09.671 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.668+0000 7ff120584700 1 --2- 172.21.15.27:0/1556930637 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1180cdc00 0x7ff1180cbe50 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff11808a900 tx=0x7ff108009780 comp rx=0 tx=0).stop 2023-10-06T01:42:09.671 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.668+0000 7ff120584700 1 -- 172.21.15.27:0/1556930637 shutdown_connections 2023-10-06T01:42:09.672 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.668+0000 7ff120584700 1 --2- 172.21.15.27:0/1556930637 >> v2:172.21.15.124:3300/0 conn(0x7ff1180c6dc0 0x7ff1180ae5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.672 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.668+0000 7ff120584700 1 --2- 172.21.15.27:0/1556930637 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1180cdc00 0x7ff1180cbe50 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.672 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.668+0000 7ff120584700 1 -- 172.21.15.27:0/1556930637 >> 172.21.15.27:0/1556930637 conn(0x7ff118099990 msgr2=0x7ff1180b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:09.672 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.669+0000 7ff120584700 1 -- 172.21.15.27:0/1556930637 shutdown_connections 2023-10-06T01:42:09.672 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.669+0000 7ff120584700 1 -- 172.21.15.27:0/1556930637 wait complete. 2023-10-06T01:42:09.674 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.670+0000 7ff120584700 1 Processor -- start 2023-10-06T01:42:09.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.670+0000 7ff120584700 1 -- start start 2023-10-06T01:42:09.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.670+0000 7ff120584700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff1180c6dc0 0x7ff11809c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:09.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.670+0000 7ff120584700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff1180cdc00 0x7ff11814c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:09.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.670+0000 7ff120584700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1180d1030 0x7ff1180c8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:09.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.670+0000 7ff120584700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff11809bf20 con 0x7ff1180d1030 2023-10-06T01:42:09.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.670+0000 7ff120584700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff11809c8f0 con 0x7ff1180cdc00 2023-10-06T01:42:09.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.670+0000 7ff120584700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff11809b5f0 con 0x7ff1180c6dc0 2023-10-06T01:42:09.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.670+0000 7ff11eb21700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1180d1030 0x7ff1180c8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:09.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.670+0000 7ff11eb21700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1180d1030 0x7ff1180c8f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37556/0 (socket says 172.21.15.27:37556) 2023-10-06T01:42:09.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.670+0000 7ff11eb21700 1 -- 172.21.15.27:0/3127636521 learned_addr learned my addr 172.21.15.27:0/3127636521 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:09.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.671+0000 7ff11e320700 1 --2- 172.21.15.27:0/3127636521 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff1180c6dc0 0x7ff11809c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:09.676 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.671+0000 7ff11db1f700 1 --2- 172.21.15.27:0/3127636521 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff1180cdc00 0x7ff11814c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:09.676 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.671+0000 7ff11eb21700 1 -- 172.21.15.27:0/3127636521 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff1180c6dc0 msgr2=0x7ff11809c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:09.676 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.671+0000 7ff11eb21700 1 --2- 172.21.15.27:0/3127636521 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff1180c6dc0 0x7ff11809c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.676 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.671+0000 7ff11eb21700 1 -- 172.21.15.27:0/3127636521 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff1180cdc00 msgr2=0x7ff11814c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:09.676 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.671+0000 7ff11eb21700 1 --2- 172.21.15.27:0/3127636521 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff1180cdc00 0x7ff11814c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.676 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.671+0000 7ff11eb21700 1 -- 172.21.15.27:0/3127636521 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff108009a20 con 0x7ff1180d1030 2023-10-06T01:42:09.676 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.671+0000 7ff11eb21700 1 --2- 172.21.15.27:0/3127636521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1180d1030 0x7ff1180c8f70 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7ff11000ea50 tx=0x7ff1100091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:09.676 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.671+0000 7ff1167fc700 1 -- 172.21.15.27:0/3127636521 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff11000c920 con 0x7ff1180d1030 2023-10-06T01:42:09.676 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.671+0000 7ff1167fc700 1 -- 172.21.15.27:0/3127636521 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff11000caa0 con 0x7ff1180d1030 2023-10-06T01:42:09.676 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.671+0000 7ff120584700 1 -- 172.21.15.27:0/3127636521 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff11809b8d0 con 0x7ff1180d1030 2023-10-06T01:42:09.676 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.671+0000 7ff1167fc700 1 -- 172.21.15.27:0/3127636521 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff11000ce60 con 0x7ff1180d1030 2023-10-06T01:42:09.676 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.671+0000 7ff120584700 1 -- 172.21.15.27:0/3127636521 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff11814c700 con 0x7ff1180d1030 2023-10-06T01:42:09.677 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.672+0000 7ff120584700 1 -- 172.21.15.27:0/3127636521 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff118064050 con 0x7ff1180d1030 2023-10-06T01:42:09.677 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.672+0000 7ff1167fc700 1 -- 172.21.15.27:0/3127636521 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff110013680 con 0x7ff1180d1030 2023-10-06T01:42:09.677 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.673+0000 7ff1167fc700 1 --2- 172.21.15.27:0/3127636521 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7ff104066f30 0x7ff1040693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:09.677 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.673+0000 7ff1167fc700 1 -- 172.21.15.27:0/3127636521 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6058+0+0 (secure 0 0 0) 0x7ff110084a70 con 0x7ff1180d1030 2023-10-06T01:42:09.677 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.673+0000 7ff11e320700 1 --2- 172.21.15.27:0/3127636521 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7ff104066f30 0x7ff1040693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:09.677 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.674+0000 7ff11e320700 1 --2- 172.21.15.27:0/3127636521 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7ff104066f30 0x7ff1040693e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff1080097c0 tx=0x7ff108019000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:09.678 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.677+0000 7ff1167fc700 1 -- 172.21.15.27:0/3127636521 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff110051b80 con 0x7ff1180d1030 2023-10-06T01:42:09.918 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.916+0000 7ff120584700 1 -- 172.21.15.27:0/3127636521 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff11809c060 con 0x7ff1180d1030 2023-10-06T01:42:09.919 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.917+0000 7ff1167fc700 1 -- 172.21.15.27:0/3127636521 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff1100551a0 con 0x7ff1180d1030 2023-10-06T01:42:09.919 INFO:teuthology.orchestra.run.smithi027.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-06T01:42:09.921 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.919+0000 7ff120584700 1 -- 172.21.15.27:0/3127636521 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7ff104066f30 msgr2=0x7ff1040693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:09.921 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.919+0000 7ff120584700 1 --2- 172.21.15.27:0/3127636521 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7ff104066f30 0x7ff1040693e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7ff1080097c0 tx=0x7ff108019000 comp rx=0 tx=0).stop 2023-10-06T01:42:09.921 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.919+0000 7ff120584700 1 -- 172.21.15.27:0/3127636521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1180d1030 msgr2=0x7ff1180c8f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:09.921 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.919+0000 7ff120584700 1 --2- 172.21.15.27:0/3127636521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1180d1030 0x7ff1180c8f70 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7ff11000ea50 tx=0x7ff1100091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.921 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.919+0000 7ff120584700 1 -- 172.21.15.27:0/3127636521 shutdown_connections 2023-10-06T01:42:09.921 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.919+0000 7ff120584700 1 --2- 172.21.15.27:0/3127636521 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7ff104066f30 0x7ff1040693e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.921 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.919+0000 7ff120584700 1 --2- 172.21.15.27:0/3127636521 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff1180c6dc0 0x7ff11809c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.922 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.919+0000 7ff120584700 1 --2- 172.21.15.27:0/3127636521 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff1180d1030 0x7ff1180c8f70 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.922 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.919+0000 7ff120584700 1 --2- 172.21.15.27:0/3127636521 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff1180cdc00 0x7ff11814c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:09.922 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.919+0000 7ff120584700 1 -- 172.21.15.27:0/3127636521 >> 172.21.15.27:0/3127636521 conn(0x7ff118099990 msgr2=0x7ff1180c77a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:09.922 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.919+0000 7ff120584700 1 -- 172.21.15.27:0/3127636521 shutdown_connections 2023-10-06T01:42:09.922 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:09.919+0000 7ff120584700 1 -- 172.21.15.27:0/3127636521 wait complete. 2023-10-06T01:42:09.932 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-06T01:42:09.932 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-06T01:42:09.932 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi027.front.sepia.ceph.com 2023-10-06T01:42:09.932 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-06T01:42:09.932 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-06T01:42:09.932 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-06T01:42:09.932 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-06T01:42:09.933 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-06T01:42:09.933 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-06T01:42:09.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:42:09.933 DEBUG:teuthology.orchestra.run.smithi027:> ip addr 2023-10-06T01:42:09.948 INFO:teuthology.orchestra.run.smithi027.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-06T01:42:09.948 INFO:teuthology.orchestra.run.smithi027.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-06T01:42:09.948 INFO:teuthology.orchestra.run.smithi027.stdout: inet 127.0.0.1/8 scope host lo 2023-10-06T01:42:09.948 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:42:09.949 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 ::1/128 scope host 2023-10-06T01:42:09.949 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:42:09.949 INFO:teuthology.orchestra.run.smithi027.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-06T01:42:09.949 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c8 brd ff:ff:ff:ff:ff:ff 2023-10-06T01:42:09.949 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f0 2023-10-06T01:42:09.949 INFO:teuthology.orchestra.run.smithi027.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-06T01:42:09.949 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 0c:c4:7a:6b:fe:c9 brd ff:ff:ff:ff:ff:ff 2023-10-06T01:42:09.949 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp7s0f1 2023-10-06T01:42:09.949 INFO:teuthology.orchestra.run.smithi027.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-06T01:42:09.949 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5a brd ff:ff:ff:ff:ff:ff 2023-10-06T01:42:09.949 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f0 2023-10-06T01:42:09.949 INFO:teuthology.orchestra.run.smithi027.stdout: inet 172.21.15.27/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-06T01:42:09.950 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft 42442sec preferred_lft 42442sec 2023-10-06T01:42:09.950 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::ae1f:6bff:fef8:395a/64 scope link 2023-10-06T01:42:09.950 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:42:09.950 INFO:teuthology.orchestra.run.smithi027.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-06T01:42:09.950 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether ac:1f:6b:f8:39:5b brd ff:ff:ff:ff:ff:ff 2023-10-06T01:42:09.950 INFO:teuthology.orchestra.run.smithi027.stdout: altname enp1s0f1 2023-10-06T01:42:09.950 INFO:teuthology.orchestra.run.smithi027.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-06T01:42:09.950 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 76:16:3c:5c:63:36 brd ff:ff:ff:ff:ff:ff 2023-10-06T01:42:09.950 INFO:teuthology.orchestra.run.smithi027.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-06T01:42:09.950 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:42:09.950 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::c478:b0ff:fe1c:e43/64 scope link 2023-10-06T01:42:09.950 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:42:09.951 INFO:teuthology.orchestra.run.smithi027.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-06T01:42:09.951 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether 76:16:3c:5c:63:36 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-06T01:42:09.951 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::7416:3cff:fe5c:6336/64 scope link 2023-10-06T01:42:09.951 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:42:09.951 INFO:teuthology.orchestra.run.smithi027.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-06T01:42:09.951 INFO:teuthology.orchestra.run.smithi027.stdout: link/ether fe:97:ab:6f:49:33 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-06T01:42:09.951 INFO:teuthology.orchestra.run.smithi027.stdout: inet6 fe80::fc97:abff:fe6f:4933/64 scope link 2023-10-06T01:42:09.951 INFO:teuthology.orchestra.run.smithi027.stdout: valid_lft forever preferred_lft forever 2023-10-06T01:42:09.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:42:09.952 DEBUG:teuthology.orchestra.run.smithi027:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-06T01:42:10.013 INFO:teuthology.orchestra.run.smithi027.stdout:1 2023-10-06T01:42:10.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:42:10.014 DEBUG:teuthology.orchestra.run.smithi027:> route 2023-10-06T01:42:10.070 INFO:teuthology.orchestra.run.smithi027.stdout:Kernel IP routing table 2023-10-06T01:42:10.071 INFO:teuthology.orchestra.run.smithi027.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-06T01:42:10.071 INFO:teuthology.orchestra.run.smithi027.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-06T01:42:10.071 INFO:teuthology.orchestra.run.smithi027.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-06T01:42:10.071 INFO:teuthology.orchestra.run.smithi027.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-06T01:42:10.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:42:10.071 DEBUG:teuthology.orchestra.run.smithi027:> (cd / && exec bash -c ' 2023-10-06T01:42:10.071 DEBUG:teuthology.orchestra.run.smithi027:> set -e 2023-10-06T01:42:10.072 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-06T01:42:10.072 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-06T01:42:10.072 DEBUG:teuthology.orchestra.run.smithi027:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-06T01:42:10.072 DEBUG:teuthology.orchestra.run.smithi027:> ') 2023-10-06T01:42:10.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:42:10.173 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2023-10-06T01:42:10.188 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-06T01:42:10.188 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-06T01:42:10.188 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-06T01:42:10.188 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:42:10.188 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link set brx.0 up 2023-10-06T01:42:10.252 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T01:42:10.253 DEBUG:teuthology.orchestra.run.smithi027:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-06T01:42:10.308 INFO:teuthology.orchestra.run.smithi027.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-06T01:42:10.308 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-06T01:42:10.308 DEBUG:teuthology.orchestra.run.smithi027:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-06T01:42:10.363 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-06T01:42:10.363 INFO:teuthology.orchestra.run:Running command with timeout 1800 2023-10-06T01:42:10.363 DEBUG:teuthology.orchestra.run.smithi027:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-06T01:42:10.444 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-06T01:42:10.444 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-06T01:42:10.444 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2023-10-06T01:42:10.452 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-06T01:42:10.452 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-06T01:42:10.453 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2023-10-06T01:42:10.453 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.124:3300,172.21.15.124:3301" 2023-10-06T01:42:10.453 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-06T01:42:10.475 INFO:teuthology.orchestra.run.smithi027.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-06T01:42:10.475 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-06T01:42:10.475 INFO:teuthology.orchestra.run.smithi027.stdout:invalid new device string format 2023-10-06T01:42:10.475 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:3300,172.21.15.124:3300,172.21.15.124:3301" 2023-10-06T01:42:10.475 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-06T01:42:10.476 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: resolved to: "172.21.15.27:6789,172.21.15.124:6789,172.21.15.124:6790" 2023-10-06T01:42:10.476 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.27:6789,172.21.15.124:6789,172.21.15.124:6790:/ 2023-10-06T01:42:10.476 INFO:teuthology.orchestra.run.smithi027.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=20a7bff0-74f8-4f2f-b6c6-899e804bc965" will pass to kernel 2023-10-06T01:42:10.478 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-06T01:42:10.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:42:10.479 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-06T01:42:10.545 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_symlink_recovery false 2023-10-06T01:42:10.709 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.707+0000 7fea37a1a700 1 Processor -- start 2023-10-06T01:42:10.709 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.708+0000 7fea37a1a700 1 -- start start 2023-10-06T01:42:10.711 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.708+0000 7fea37a1a700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fea301598d0 0x7fea3015d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:10.711 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.708+0000 7fea37a1a700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fea3015d8e0 0x7fea300cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:10.711 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.708+0000 7fea37a1a700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea3015dd20 con 0x7fea30159500 2023-10-06T01:42:10.711 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.708+0000 7fea37a1a700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea300b1110 con 0x7fea301598d0 2023-10-06T01:42:10.712 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.708+0000 7fea37a1a700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea300b13b0 con 0x7fea3015d8e0 2023-10-06T01:42:10.712 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.709+0000 7fea34fb5700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fea301598d0 0x7fea3015d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:10.712 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.709+0000 7fea35fb7700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fea3015d8e0 0x7fea300cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:10.712 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.709+0000 7fea357b6700 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fea30159500 0x7fea300b3fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.27:38002/0 (socket says 172.21.15.27:38002) 2023-10-06T01:42:10.712 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.709+0000 7fea34fb5700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fea301598d0 0x7fea3015d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.27:56132/0 (socket says 172.21.15.27:56132) 2023-10-06T01:42:10.712 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.709+0000 7fea35fb7700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fea3015d8e0 0x7fea300cf6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.27:40836/0 (socket says 172.21.15.27:40836) 2023-10-06T01:42:10.712 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.709+0000 7fea357b6700 1 -- 172.21.15.27:0/2107304408 learned_addr learned my addr 172.21.15.27:0/2107304408 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:10.712 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.709+0000 7fea2ffff700 1 -- 172.21.15.27:0/2107304408 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1188166742 0 0) 0x7fea3015dd20 con 0x7fea30159500 2023-10-06T01:42:10.712 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.709+0000 7fea2ffff700 1 -- 172.21.15.27:0/2107304408 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea18003570 con 0x7fea30159500 2023-10-06T01:42:10.712 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.709+0000 7fea34fb5700 1 -- 172.21.15.27:0/2107304408 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fea3015d8e0 msgr2=0x7fea300cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:10.712 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.709+0000 7fea34fb5700 1 --2- 172.21.15.27:0/2107304408 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fea3015d8e0 0x7fea300cf6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:10.712 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.709+0000 7fea34fb5700 1 -- 172.21.15.27:0/2107304408 >> v1:172.21.15.124:6789/0 conn(0x7fea30159500 legacy=0x7fea300b3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:10.712 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.709+0000 7fea34fb5700 1 -- 172.21.15.27:0/2107304408 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea300b1690 con 0x7fea301598d0 2023-10-06T01:42:10.713 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.709+0000 7fea35fb7700 1 --2- 172.21.15.27:0/2107304408 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fea3015d8e0 0x7fea300cf6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:42:10.713 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.709+0000 7fea34fb5700 1 --2- 172.21.15.27:0/2107304408 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fea301598d0 0x7fea3015d3a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fea240098e0 tx=0x7fea240091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=994724aab6568333 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:10.713 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.709+0000 7fea2ffff700 1 -- 172.21.15.27:0/2107304408 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea24011630 con 0x7fea301598d0 2023-10-06T01:42:10.713 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.709+0000 7fea2ffff700 1 -- 172.21.15.27:0/2107304408 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea24011790 con 0x7fea301598d0 2023-10-06T01:42:10.713 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.709+0000 7fea2ffff700 1 -- 172.21.15.27:0/2107304408 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea24011b30 con 0x7fea301598d0 2023-10-06T01:42:10.713 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.710+0000 7fea37a1a700 1 -- 172.21.15.27:0/2107304408 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fea301598d0 msgr2=0x7fea3015d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:10.713 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.710+0000 7fea37a1a700 1 --2- 172.21.15.27:0/2107304408 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fea301598d0 0x7fea3015d3a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fea240098e0 tx=0x7fea240091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:10.713 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.710+0000 7fea37a1a700 1 -- 172.21.15.27:0/2107304408 shutdown_connections 2023-10-06T01:42:10.713 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.710+0000 7fea37a1a700 1 --2- 172.21.15.27:0/2107304408 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fea3015d8e0 0x7fea300cf6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:10.713 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.710+0000 7fea37a1a700 1 --2- 172.21.15.27:0/2107304408 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fea301598d0 0x7fea3015d3a0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:10.713 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.710+0000 7fea37a1a700 1 -- 172.21.15.27:0/2107304408 >> 172.21.15.27:0/2107304408 conn(0x7fea30067630 msgr2=0x7fea30069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:10.713 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.710+0000 7fea37a1a700 1 -- 172.21.15.27:0/2107304408 shutdown_connections 2023-10-06T01:42:10.714 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.710+0000 7fea37a1a700 1 -- 172.21.15.27:0/2107304408 wait complete. 2023-10-06T01:42:10.716 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.711+0000 7fea37a1a700 1 Processor -- start 2023-10-06T01:42:10.716 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.711+0000 7fea37a1a700 1 -- start start 2023-10-06T01:42:10.716 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.712+0000 7fea37a1a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea30159500 0x7fea3014f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:10.716 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.712+0000 7fea37a1a700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fea301598d0 0x7fea3015a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:10.716 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.712+0000 7fea37a1a700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fea3015d8e0 0x7fea3015f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:10.716 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.712+0000 7fea37a1a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea3015bfd0 con 0x7fea30159500 2023-10-06T01:42:10.717 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.712+0000 7fea37a1a700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea3015acd0 con 0x7fea301598d0 2023-10-06T01:42:10.717 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.712+0000 7fea37a1a700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea3015af30 con 0x7fea3015d8e0 2023-10-06T01:42:10.717 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.712+0000 7fea357b6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea30159500 0x7fea3014f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:10.717 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.712+0000 7fea357b6700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea30159500 0x7fea3014f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37568/0 (socket says 172.21.15.27:37568) 2023-10-06T01:42:10.717 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.712+0000 7fea357b6700 1 -- 172.21.15.27:0/1414425325 learned_addr learned my addr 172.21.15.27:0/1414425325 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:10.717 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.712+0000 7fea34fb5700 1 --2- 172.21.15.27:0/1414425325 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fea301598d0 0x7fea3015a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:10.717 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.712+0000 7fea35fb7700 1 --2- 172.21.15.27:0/1414425325 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fea3015d8e0 0x7fea3015f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:10.717 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.712+0000 7fea357b6700 1 -- 172.21.15.27:0/1414425325 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fea3015d8e0 msgr2=0x7fea3015f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:10.717 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.712+0000 7fea357b6700 1 --2- 172.21.15.27:0/1414425325 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fea3015d8e0 0x7fea3015f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:10.717 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.712+0000 7fea357b6700 1 -- 172.21.15.27:0/1414425325 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fea301598d0 msgr2=0x7fea3015a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:10.717 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.712+0000 7fea357b6700 1 --2- 172.21.15.27:0/1414425325 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fea301598d0 0x7fea3015a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:10.718 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.712+0000 7fea357b6700 1 -- 172.21.15.27:0/1414425325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea20009a20 con 0x7fea30159500 2023-10-06T01:42:10.718 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.713+0000 7fea357b6700 1 --2- 172.21.15.27:0/1414425325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea30159500 0x7fea3014f110 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fea2800c7c0 tx=0x7fea28007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:10.718 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.713+0000 7fea2d7fa700 1 -- 172.21.15.27:0/1414425325 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea28005b30 con 0x7fea30159500 2023-10-06T01:42:10.718 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.713+0000 7fea2d7fa700 1 -- 172.21.15.27:0/1414425325 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea28005c90 con 0x7fea30159500 2023-10-06T01:42:10.718 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.713+0000 7fea2d7fa700 1 -- 172.21.15.27:0/1414425325 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea28007340 con 0x7fea30159500 2023-10-06T01:42:10.718 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.713+0000 7fea37a1a700 1 -- 172.21.15.27:0/1414425325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea24009420 con 0x7fea30159500 2023-10-06T01:42:10.718 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.713+0000 7fea37a1a700 1 -- 172.21.15.27:0/1414425325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea3015b7d0 con 0x7fea30159500 2023-10-06T01:42:10.718 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.714+0000 7fea37a1a700 1 -- 172.21.15.27:0/1414425325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea30064050 con 0x7fea30159500 2023-10-06T01:42:10.718 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.714+0000 7fea2d7fa700 1 -- 172.21.15.27:0/1414425325 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fea2801b400 con 0x7fea30159500 2023-10-06T01:42:10.718 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.715+0000 7fea2d7fa700 1 --2- 172.21.15.27:0/1414425325 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fea1c066f30 0x7fea1c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:10.718 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.715+0000 7fea2d7fa700 1 -- 172.21.15.27:0/1414425325 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fea280841a0 con 0x7fea30159500 2023-10-06T01:42:10.719 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.715+0000 7fea34fb5700 1 --2- 172.21.15.27:0/1414425325 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fea1c066f30 0x7fea1c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:10.719 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.715+0000 7fea34fb5700 1 --2- 172.21.15.27:0/1414425325 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fea1c066f30 0x7fea1c0693e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fea240091e0 tx=0x7fea24017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:10.720 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.719+0000 7fea2d7fa700 1 -- 172.21.15.27:0/1414425325 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fea28051330 con 0x7fea30159500 2023-10-06T01:42:10.915 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.914+0000 7fea37a1a700 1 -- 172.21.15.27:0/1414425325 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mds_symlink_recovery}] v 0) v1 -- 0x7fea300b1970 con 0x7fea30159500 2023-10-06T01:42:10.918 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.917+0000 7fea2d7fa700 1 -- 172.21.15.27:0/1414425325 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_symlink_recovery}]=0 v11)=0 v11) v1 ==== 122+0+0 (secure 0 0 0) 0x7fea28054950 con 0x7fea30159500 2023-10-06T01:42:10.921 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.919+0000 7fea37a1a700 1 -- 172.21.15.27:0/1414425325 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fea1c066f30 msgr2=0x7fea1c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:10.921 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.919+0000 7fea37a1a700 1 --2- 172.21.15.27:0/1414425325 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fea1c066f30 0x7fea1c0693e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fea240091e0 tx=0x7fea24017000 comp rx=0 tx=0).stop 2023-10-06T01:42:10.921 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.919+0000 7fea37a1a700 1 -- 172.21.15.27:0/1414425325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea30159500 msgr2=0x7fea3014f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:10.921 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.919+0000 7fea37a1a700 1 --2- 172.21.15.27:0/1414425325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea30159500 0x7fea3014f110 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fea2800c7c0 tx=0x7fea28007e00 comp rx=0 tx=0).stop 2023-10-06T01:42:10.921 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.919+0000 7fea37a1a700 1 -- 172.21.15.27:0/1414425325 shutdown_connections 2023-10-06T01:42:10.921 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.919+0000 7fea37a1a700 1 --2- 172.21.15.27:0/1414425325 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fea1c066f30 0x7fea1c0693e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:10.921 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.919+0000 7fea37a1a700 1 --2- 172.21.15.27:0/1414425325 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fea3015d8e0 0x7fea3015f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:10.921 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.919+0000 7fea37a1a700 1 --2- 172.21.15.27:0/1414425325 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fea30159500 0x7fea3014f110 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:10.921 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.919+0000 7fea37a1a700 1 --2- 172.21.15.27:0/1414425325 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fea301598d0 0x7fea3015a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:10.921 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.919+0000 7fea37a1a700 1 -- 172.21.15.27:0/1414425325 >> 172.21.15.27:0/1414425325 conn(0x7fea30067630 msgr2=0x7fea300b1d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:10.921 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.919+0000 7fea37a1a700 1 -- 172.21.15.27:0/1414425325 shutdown_connections 2023-10-06T01:42:10.922 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:10.919+0000 7fea37a1a700 1 -- 172.21.15.27:0/1414425325 wait complete. 2023-10-06T01:42:10.932 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T01:42:10.933 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir sym_dir0) 2023-10-06T01:42:10.952 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T01:42:10.953 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch sym_dir0/file1) 2023-10-06T01:42:11.009 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T01:42:11.009 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln -s sym_dir0/file1 sym_dir0/symlink_file1) 2023-10-06T01:42:11.065 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits daemon-helper kill python3 -c ' 2023-10-06T01:42:11.065 DEBUG:teuthology.orchestra.run.smithi027:> import os 2023-10-06T01:42:11.065 DEBUG:teuthology.orchestra.run.smithi027:> import stat 2023-10-06T01:42:11.065 DEBUG:teuthology.orchestra.run.smithi027:> 2023-10-06T01:42:11.065 DEBUG:teuthology.orchestra.run.smithi027:> print(os.lstat("/home/ubuntu/cephtest/mnt.0/sym_dir0/symlink_file1").st_ino) 2023-10-06T01:42:11.066 DEBUG:teuthology.orchestra.run.smithi027:> ' 2023-10-06T01:42:11.170 INFO:teuthology.orchestra.run.smithi027.stdout:1099511628278 2023-10-06T01:42:11.359 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T01:42:11.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.518+0000 7f993b334700 1 Processor -- start 2023-10-06T01:42:11.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.518+0000 7f993b334700 1 -- start start 2023-10-06T01:42:11.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.519+0000 7f993b334700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f993415ce30 0x7f9934164790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:11.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.519+0000 7f993b334700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99340af040 con 0x7f99340cbda0 2023-10-06T01:42:11.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.519+0000 7f993b334700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99340afd70 con 0x7f99340cc170 2023-10-06T01:42:11.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.519+0000 7f993b334700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f993415d290 con 0x7f993415ce30 2023-10-06T01:42:11.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.519+0000 7f993a332700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f99340cbda0 0x7f9934067630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52138/0 (socket says 172.21.15.27:52138) 2023-10-06T01:42:11.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.519+0000 7f993a332700 1 -- 172.21.15.27:0/3461149104 learned_addr learned my addr 172.21.15.27:0/3461149104 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:11.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.519+0000 7f993ab33700 1 --2- 172.21.15.27:0/3461149104 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f993415ce30 0x7f9934164790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:11.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.519+0000 7f9939330700 1 -- 172.21.15.27:0/3461149104 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4097310772 0 0) 0x7f99340af040 con 0x7f99340cbda0 2023-10-06T01:42:11.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.519+0000 7f9939330700 1 -- 172.21.15.27:0/3461149104 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f991c003570 con 0x7f99340cbda0 2023-10-06T01:42:11.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.520+0000 7f9939330700 1 -- 172.21.15.27:0/3461149104 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3074349282 0 0) 0x7f99340afd70 con 0x7f99340cc170 2023-10-06T01:42:11.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.520+0000 7f9939330700 1 -- 172.21.15.27:0/3461149104 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99340af040 con 0x7f99340cc170 2023-10-06T01:42:11.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.520+0000 7f9939330700 1 -- 172.21.15.27:0/3461149104 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1995904238 0 0) 0x7f991c003570 con 0x7f99340cbda0 2023-10-06T01:42:11.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.520+0000 7f9939330700 1 -- 172.21.15.27:0/3461149104 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f993415ce30 msgr2=0x7f9934164790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:11.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.520+0000 7f9939330700 1 --2- 172.21.15.27:0/3461149104 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f993415ce30 0x7f9934164790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:11.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.520+0000 7f9939330700 1 -- 172.21.15.27:0/3461149104 >> v1:172.21.15.124:6789/0 conn(0x7f99340cc170 legacy=0x7f993415c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:11.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.520+0000 7f9939330700 1 -- 172.21.15.27:0/3461149104 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9934068080 con 0x7f99340cbda0 2023-10-06T01:42:11.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.520+0000 7f9939330700 1 -- 172.21.15.27:0/3461149104 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2619078944 0 0) 0x7f992c001470 con 0x7f99340cbda0 2023-10-06T01:42:11.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.520+0000 7f9939330700 1 -- 172.21.15.27:0/3461149104 >> v1:172.21.15.27:6789/0 conn(0x7f99340cbda0 legacy=0x7f9934067630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:11.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.520+0000 7f9939330700 1 --2- 172.21.15.27:0/3461149104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f991c003c20 0x7f991c004010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:11.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.520+0000 7f9939330700 1 --2- 172.21.15.27:0/3461149104 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f993415ce30 0x7f991c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:11.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.520+0000 7f9939330700 1 --2- 172.21.15.27:0/3461149104 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f991c004d50 0x7f991c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:11.525 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.520+0000 7f9939330700 1 -- 172.21.15.27:0/3461149104 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991c00bbf0 con 0x7f991c003c20 2023-10-06T01:42:11.525 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.520+0000 7f9939330700 1 -- 172.21.15.27:0/3461149104 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f991c00bd30 con 0x7f991c004d50 2023-10-06T01:42:11.525 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.520+0000 7f9939330700 1 -- 172.21.15.27:0/3461149104 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f991c00bea0 con 0x7f993415ce30 2023-10-06T01:42:11.525 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.520+0000 7f993a332700 1 --2- 172.21.15.27:0/3461149104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f991c003c20 0x7f991c004010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:11.525 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.520+0000 7f9939b31700 1 --2- 172.21.15.27:0/3461149104 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f993415ce30 0x7f991c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:11.525 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.520+0000 7f993ab33700 1 --2- 172.21.15.27:0/3461149104 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f991c004d50 0x7f991c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:11.525 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.520+0000 7f993a332700 1 -- 172.21.15.27:0/3461149104 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f993415ce30 msgr2=0x7f991c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:11.525 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.520+0000 7f993a332700 1 --2- 172.21.15.27:0/3461149104 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f993415ce30 0x7f991c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:11.525 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.520+0000 7f993a332700 1 -- 172.21.15.27:0/3461149104 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f991c004d50 msgr2=0x7f991c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:11.525 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.520+0000 7f993a332700 1 --2- 172.21.15.27:0/3461149104 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f991c004d50 0x7f991c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:11.525 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.520+0000 7f993a332700 1 -- 172.21.15.27:0/3461149104 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f991c003570 con 0x7f991c003c20 2023-10-06T01:42:11.525 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.520+0000 7f993a332700 1 --2- 172.21.15.27:0/3461149104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f991c003c20 0x7f991c004010 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f992c00c1e0 tx=0x7f992c001d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43b9167349fea24e server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:11.525 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.520+0000 7f9939330700 1 -- 172.21.15.27:0/3461149104 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f992c00cbe0 con 0x7f991c003c20 2023-10-06T01:42:11.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.520+0000 7f9939330700 1 -- 172.21.15.27:0/3461149104 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f992c00cd40 con 0x7f991c003c20 2023-10-06T01:42:11.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.520+0000 7f9939330700 1 -- 172.21.15.27:0/3461149104 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f992c005b70 con 0x7f991c003c20 2023-10-06T01:42:11.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.521+0000 7f993b334700 1 -- 172.21.15.27:0/3461149104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f991c003c20 msgr2=0x7f991c004010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:11.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.521+0000 7f993b334700 1 --2- 172.21.15.27:0/3461149104 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f991c003c20 0x7f991c004010 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f992c00c1e0 tx=0x7f992c001d40 comp rx=0 tx=0).stop 2023-10-06T01:42:11.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.521+0000 7f993ab33700 1 -- 172.21.15.27:0/3461149104 reap_dead start 2023-10-06T01:42:11.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.521+0000 7f993b334700 1 -- 172.21.15.27:0/3461149104 shutdown_connections 2023-10-06T01:42:11.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.521+0000 7f993b334700 1 -- 172.21.15.27:0/3461149104 >> 172.21.15.27:0/3461149104 conn(0x7f99340d0f70 msgr2=0x7f99340ccf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:11.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.521+0000 7f993b334700 1 -- 172.21.15.27:0/3461149104 shutdown_connections 2023-10-06T01:42:11.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.521+0000 7f993b334700 1 -- 172.21.15.27:0/3461149104 wait complete. 2023-10-06T01:42:11.529 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.522+0000 7f993b334700 1 Processor -- start 2023-10-06T01:42:11.529 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.522+0000 7f993b334700 1 -- start start 2023-10-06T01:42:11.529 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.523+0000 7f993b334700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f991c003c20 0x7f9934156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:11.529 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.523+0000 7f993b334700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f99340b26c0 0x7f99340b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:11.529 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.523+0000 7f993b334700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f99340b32a0 0x7f99340b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:11.529 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.523+0000 7f993b334700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99340b57a0 con 0x7f991c003c20 2023-10-06T01:42:11.529 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.523+0000 7f993b334700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99340b3c30 con 0x7f99340b32a0 2023-10-06T01:42:11.529 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.523+0000 7f993b334700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99340b4bd0 con 0x7f99340b26c0 2023-10-06T01:42:11.529 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.523+0000 7f993a332700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f991c003c20 0x7f9934156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:11.530 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.523+0000 7f993a332700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f991c003c20 0x7f9934156ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37580/0 (socket says 172.21.15.27:37580) 2023-10-06T01:42:11.530 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.523+0000 7f993a332700 1 -- 172.21.15.27:0/3395841718 learned_addr learned my addr 172.21.15.27:0/3395841718 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:11.530 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.523+0000 7f9939b31700 1 --2- 172.21.15.27:0/3395841718 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f99340b26c0 0x7f99340b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:11.530 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.523+0000 7f993ab33700 1 --2- 172.21.15.27:0/3395841718 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f99340b32a0 0x7f99340b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:11.530 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.523+0000 7f993a332700 1 -- 172.21.15.27:0/3395841718 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f99340b26c0 msgr2=0x7f99340b2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:11.530 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.523+0000 7f993a332700 1 --2- 172.21.15.27:0/3395841718 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f99340b26c0 0x7f99340b2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:11.530 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.523+0000 7f993a332700 1 -- 172.21.15.27:0/3395841718 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f99340b32a0 msgr2=0x7f99340b36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:11.530 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.523+0000 7f993a332700 1 --2- 172.21.15.27:0/3395841718 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f99340b32a0 0x7f99340b36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:11.530 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.523+0000 7f993a332700 1 -- 172.21.15.27:0/3395841718 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99340b4e70 con 0x7f991c003c20 2023-10-06T01:42:11.530 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.523+0000 7f993ab33700 1 --2- 172.21.15.27:0/3395841718 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f99340b32a0 0x7f99340b36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:42:11.530 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.524+0000 7f993a332700 1 --2- 172.21.15.27:0/3395841718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f991c003c20 0x7f9934156ed0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f992c00c7d0 tx=0x7f992c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:11.530 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.524+0000 7f99327fc700 1 -- 172.21.15.27:0/3395841718 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f992c00cbe0 con 0x7f991c003c20 2023-10-06T01:42:11.531 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.524+0000 7f99327fc700 1 -- 172.21.15.27:0/3395841718 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f992c00ceb0 con 0x7f991c003c20 2023-10-06T01:42:11.531 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.524+0000 7f993b334700 1 -- 172.21.15.27:0/3395841718 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99340b5150 con 0x7f991c003c20 2023-10-06T01:42:11.531 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.524+0000 7f99327fc700 1 -- 172.21.15.27:0/3395841718 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f992c007510 con 0x7f991c003c20 2023-10-06T01:42:11.531 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.524+0000 7f993b334700 1 -- 172.21.15.27:0/3395841718 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99340b5c30 con 0x7f991c003c20 2023-10-06T01:42:11.531 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.525+0000 7f993b334700 1 -- 172.21.15.27:0/3395841718 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f993404c3c0 con 0x7f991c003c20 2023-10-06T01:42:11.531 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.525+0000 7f99327fc700 1 -- 172.21.15.27:0/3395841718 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f992c019030 con 0x7f991c003c20 2023-10-06T01:42:11.531 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.525+0000 7f99327fc700 1 --2- 172.21.15.27:0/3395841718 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9920066f30 0x7f99200693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:11.531 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.526+0000 7f99327fc700 1 -- 172.21.15.27:0/3395841718 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f992c085870 con 0x7f991c003c20 2023-10-06T01:42:11.531 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.526+0000 7f9939b31700 1 --2- 172.21.15.27:0/3395841718 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9920066f30 0x7f99200693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:11.531 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.526+0000 7f9939b31700 1 --2- 172.21.15.27:0/3395841718 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9920066f30 0x7f99200693e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f99280010e0 tx=0x7f9928007d30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:11.532 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.530+0000 7f99327fc700 1 -- 172.21.15.27:0/3395841718 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f992c052980 con 0x7f991c003c20 2023-10-06T01:42:11.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.771+0000 7f993b334700 1 -- 172.21.15.27:0/3395841718 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99340cdc00 con 0x7f991c003c20 2023-10-06T01:42:11.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.771+0000 7f99327fc700 1 -- 172.21.15.27:0/3395841718 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4597 (secure 0 0 0) 0x7f992c017090 con 0x7f991c003c20 2023-10-06T01:42:11.773 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:11.773 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4870,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6836/1043044736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":1043044736},{"type":"v1","addr":"172.21.15.124:6836","nonce":1043044736}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4962,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/1207987117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1207987117},{"type":"v1","addr":"172.21.15.27:6834","nonce":1207987117}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4965,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/943261840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":943261840},{"type":"v1","addr":"172.21.15.27:6835","nonce":943261840}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T01:42:03.036784+0000","modified":"2023-10-06T01:42:06.652163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4974},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4974":{"gid":4974,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.124:6837/325247112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6835","nonce":325247112},{"type":"v1","addr":"172.21.15.124:6837","nonce":325247112}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-10-06T01:42:11.775 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.773+0000 7f993b334700 1 -- 172.21.15.27:0/3395841718 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9920066f30 msgr2=0x7f99200693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:11.775 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.773+0000 7f993b334700 1 --2- 172.21.15.27:0/3395841718 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9920066f30 0x7f99200693e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f99280010e0 tx=0x7f9928007d30 comp rx=0 tx=0).stop 2023-10-06T01:42:11.775 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.773+0000 7f993b334700 1 -- 172.21.15.27:0/3395841718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f991c003c20 msgr2=0x7f9934156ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:11.775 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.773+0000 7f993b334700 1 --2- 172.21.15.27:0/3395841718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f991c003c20 0x7f9934156ed0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f992c00c7d0 tx=0x7f992c015000 comp rx=0 tx=0).stop 2023-10-06T01:42:11.775 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.773+0000 7f993b334700 1 -- 172.21.15.27:0/3395841718 shutdown_connections 2023-10-06T01:42:11.775 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.773+0000 7f993b334700 1 --2- 172.21.15.27:0/3395841718 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9920066f30 0x7f99200693e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:11.775 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.773+0000 7f993b334700 1 --2- 172.21.15.27:0/3395841718 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f99340b32a0 0x7f99340b36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:11.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.773+0000 7f993b334700 1 --2- 172.21.15.27:0/3395841718 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f991c003c20 0x7f9934156ed0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:11.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.773+0000 7f993b334700 1 --2- 172.21.15.27:0/3395841718 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f99340b26c0 0x7f99340b2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:11.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.773+0000 7f993b334700 1 -- 172.21.15.27:0/3395841718 >> 172.21.15.27:0/3395841718 conn(0x7f99340d0f70 msgr2=0x7f99340cf480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:11.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.773+0000 7f993b334700 1 -- 172.21.15.27:0/3395841718 shutdown_connections 2023-10-06T01:42:11.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.773+0000 7f993b334700 1 -- 172.21.15.27:0/3395841718 wait complete. 2023-10-06T01:42:11.776 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 15 2023-10-06T01:42:11.787 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json flush journal 2023-10-06T01:42:11.936 INFO:teuthology.orchestra.run.smithi124.stdout:{"message":"","return_code":0} 2023-10-06T01:42:11.936 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2023-10-06T01:42:11.936 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001f6.00000000 symlink 2023-10-06T01:42:11.976 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.975+0000 7f8e6e913000 1 Processor -- start 2023-10-06T01:42:11.976 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.975+0000 7f8e6e913000 1 -- start start 2023-10-06T01:42:11.979 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.976+0000 7f8e6e913000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e54002330 0x55f8192794a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:11.979 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.976+0000 7f8e6e913000 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55f81927cd40 0x55f8191d6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:11.979 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.976+0000 7f8e6e913000 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55f81927c7b0 0x55f8191d6d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:11.979 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.976+0000 7f8e6e913000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f819153320 con 0x7f8e54002330 2023-10-06T01:42:11.979 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.976+0000 7f8e6e913000 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8190f7980 con 0x55f81927c7b0 2023-10-06T01:42:11.979 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.976+0000 7f8e6e913000 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f8190f7620 con 0x55f81927cd40 2023-10-06T01:42:11.979 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.976+0000 7f8e67b4e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e54002330 0x55f8192794a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:11.979 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.976+0000 7f8e67b4e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e54002330 0x55f8192794a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37602/0 (socket says 172.21.15.27:37602) 2023-10-06T01:42:11.979 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.976+0000 7f8e6734d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55f81927cd40 0x55f8191d6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:11.980 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.976+0000 7f8e67b4e700 1 -- 172.21.15.27:0/1322415709 learned_addr learned my addr 172.21.15.27:0/1322415709 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:11.980 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.976+0000 7f8e6834f700 1 --2- 172.21.15.27:0/1322415709 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55f81927c7b0 0x55f8191d6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:11.980 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.976+0000 7f8e6734d700 1 -- 172.21.15.27:0/1322415709 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55f81927c7b0 msgr2=0x55f8191d6d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:11.980 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.976+0000 7f8e6734d700 1 --2- 172.21.15.27:0/1322415709 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55f81927c7b0 0x55f8191d6d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:11.980 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.976+0000 7f8e6734d700 1 -- 172.21.15.27:0/1322415709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e54002330 msgr2=0x55f8192794a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:11.980 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.976+0000 7f8e6734d700 1 --2- 172.21.15.27:0/1322415709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e54002330 0x55f8192794a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:11.980 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.976+0000 7f8e6734d700 1 -- 172.21.15.27:0/1322415709 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f8190f9ae0 con 0x55f81927cd40 2023-10-06T01:42:11.980 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.976+0000 7f8e6834f700 1 --2- 172.21.15.27:0/1322415709 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55f81927c7b0 0x55f8191d6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:42:11.980 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.976+0000 7f8e67b4e700 1 --2- 172.21.15.27:0/1322415709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e54002330 0x55f8192794a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:42:11.980 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.976+0000 7f8e6734d700 1 --2- 172.21.15.27:0/1322415709 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55f81927cd40 0x55f8191d6810 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8e50002ee0 tx=0x55f8191d7e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec80ace135ace0c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:11.980 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.976+0000 7f8e6634b700 1 -- 172.21.15.27:0/1322415709 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e50009a60 con 0x55f81927cd40 2023-10-06T01:42:11.980 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.976+0000 7f8e6634b700 1 -- 172.21.15.27:0/1322415709 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e5000a040 con 0x55f81927cd40 2023-10-06T01:42:11.981 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.976+0000 7f8e6634b700 1 -- 172.21.15.27:0/1322415709 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e50009e30 con 0x55f81927cd40 2023-10-06T01:42:11.981 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.977+0000 7f8e6e913000 1 -- 172.21.15.27:0/1322415709 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55f81927cd40 msgr2=0x55f8191d6810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:11.981 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.977+0000 7f8e6e913000 1 --2- 172.21.15.27:0/1322415709 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55f81927cd40 0x55f8191d6810 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8e50002ee0 tx=0x55f8191d7e60 comp rx=0 tx=0).stop 2023-10-06T01:42:11.981 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.977+0000 7f8e6e913000 1 -- 172.21.15.27:0/1322415709 shutdown_connections 2023-10-06T01:42:11.981 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.977+0000 7f8e6e913000 1 --2- 172.21.15.27:0/1322415709 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55f81927c7b0 0x55f8191d6d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:11.981 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.977+0000 7f8e6e913000 1 --2- 172.21.15.27:0/1322415709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e54002330 0x55f8192794a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:11.981 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.977+0000 7f8e6e913000 1 --2- 172.21.15.27:0/1322415709 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55f81927cd40 0x55f8191d6810 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:11.981 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.977+0000 7f8e6e913000 1 -- 172.21.15.27:0/1322415709 >> 172.21.15.27:0/1322415709 conn(0x55f8191ad8a0 msgr2=0x55f8191adca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:11.981 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.977+0000 7f8e6e913000 1 -- 172.21.15.27:0/1322415709 shutdown_connections 2023-10-06T01:42:11.981 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.977+0000 7f8e6e913000 1 -- 172.21.15.27:0/1322415709 wait complete. 2023-10-06T01:42:11.981 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.977+0000 7f8e6e913000 1 Processor -- start 2023-10-06T01:42:11.982 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.978+0000 7f8e6e913000 1 -- start start 2023-10-06T01:42:11.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T01:42:11.986 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.978+0000 7f8e6e913000 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55f81927c7b0 0x55f8191f21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:11.986 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.978+0000 7f8e6e913000 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55f81927cd40 0x55f8191d6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:11.986 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.978+0000 7f8e6e913000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e54002330 0x55f8192775d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:11.986 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.978+0000 7f8e6e913000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8190ee400 con 0x7f8e54002330 2023-10-06T01:42:11.986 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.978+0000 7f8e6e913000 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8191d6ff0 con 0x55f81927cd40 2023-10-06T01:42:11.986 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.978+0000 7f8e6e913000 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f8191f2ac0 con 0x55f81927c7b0 2023-10-06T01:42:11.986 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.978+0000 7f8e6834f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e54002330 0x55f8192775d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:11.986 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.978+0000 7f8e6834f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e54002330 0x55f8192775d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37618/0 (socket says 172.21.15.27:37618) 2023-10-06T01:42:11.986 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.978+0000 7f8e6834f700 1 -- 172.21.15.27:0/159494482 learned_addr learned my addr 172.21.15.27:0/159494482 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:11.986 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.979+0000 7f8e6734d700 1 --2- 172.21.15.27:0/159494482 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55f81927cd40 0x55f8191d6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:11.987 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.979+0000 7f8e67b4e700 1 --2- 172.21.15.27:0/159494482 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55f81927c7b0 0x55f8191f21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:11.987 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.979+0000 7f8e6834f700 1 -- 172.21.15.27:0/159494482 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55f81927c7b0 msgr2=0x55f8191f21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:11.987 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.979+0000 7f8e6834f700 1 --2- 172.21.15.27:0/159494482 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55f81927c7b0 0x55f8191f21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:11.987 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.979+0000 7f8e6834f700 1 -- 172.21.15.27:0/159494482 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55f81927cd40 msgr2=0x55f8191d6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:11.987 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.979+0000 7f8e6834f700 1 --2- 172.21.15.27:0/159494482 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55f81927cd40 0x55f8191d6ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:11.987 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.979+0000 7f8e6834f700 1 -- 172.21.15.27:0/159494482 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e5800a390 con 0x7f8e54002330 2023-10-06T01:42:11.987 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.979+0000 7f8e67b4e700 1 --2- 172.21.15.27:0/159494482 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55f81927c7b0 0x55f8191f21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:42:11.987 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.979+0000 7f8e6734d700 1 --2- 172.21.15.27:0/159494482 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55f81927cd40 0x55f8191d6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:42:11.987 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.979+0000 7f8e6834f700 1 --2- 172.21.15.27:0/159494482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e54002330 0x55f8192775d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x55f8191dcbc0 tx=0x7f8e58007040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:11.987 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.979+0000 7f8e64b48700 1 -- 172.21.15.27:0/159494482 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e580144f0 con 0x7f8e54002330 2023-10-06T01:42:11.987 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.979+0000 7f8e64b48700 1 -- 172.21.15.27:0/159494482 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e58018070 con 0x7f8e54002330 2023-10-06T01:42:11.987 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.979+0000 7f8e64b48700 1 -- 172.21.15.27:0/159494482 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e580148c0 con 0x7f8e54002330 2023-10-06T01:42:11.988 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.979+0000 7f8e6e913000 1 -- 172.21.15.27:0/159494482 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e54001e10 con 0x7f8e54002330 2023-10-06T01:42:11.988 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.979+0000 7f8e6e913000 1 -- 172.21.15.27:0/159494482 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f8191063d0 con 0x7f8e54002330 2023-10-06T01:42:11.988 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.980+0000 7f8e64b48700 1 -- 172.21.15.27:0/159494482 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f8e5801a030 con 0x7f8e54002330 2023-10-06T01:42:11.988 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.981+0000 7f8e64b48700 1 --2- 172.21.15.27:0/159494482 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8e48065890 0x7f8e48067d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:11.988 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.981+0000 7f8e64b48700 1 -- 172.21.15.27:0/159494482 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f8e58083bc0 con 0x7f8e54002330 2023-10-06T01:42:11.988 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.981+0000 7f8e67b4e700 1 --2- 172.21.15.27:0/159494482 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8e48065890 0x7f8e48067d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:11.988 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.981+0000 7f8e6e913000 1 --2- 172.21.15.27:0/159494482 >> [v2:172.21.15.124:6800/90863,v1:172.21.15.124:6801/90863] conn(0x55f8191da6a0 0x55f8191f0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:11.988 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.981+0000 7f8e6e913000 1 -- 172.21.15.27:0/159494482 --> [v2:172.21.15.124:6800/90863,v1:172.21.15.124:6801/90863] -- osd_op(unknown.0.0:1 6.2a 6:57fe7535:::100000001f6.00000000:head [getxattr symlink in=7b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) v8 -- 0x55f8191f0fb0 con 0x55f8191da6a0 2023-10-06T01:42:11.988 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.981+0000 7f8e6734d700 1 --2- 172.21.15.27:0/159494482 >> [v2:172.21.15.124:6800/90863,v1:172.21.15.124:6801/90863] conn(0x55f8191da6a0 0x55f8191f0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:11.988 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.981+0000 7f8e67b4e700 1 --2- 172.21.15.27:0/159494482 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8e48065890 0x7f8e48067d40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8e60001160 tx=0x7f8e60016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:11.988 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.981+0000 7f8e6734d700 1 --2- 172.21.15.27:0/159494482 >> [v2:172.21.15.124:6800/90863,v1:172.21.15.124:6801/90863] conn(0x55f8191da6a0 0x55f8191f0a70 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:11.988 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.982+0000 7f8e6734d700 1 -- 172.21.15.27:0/159494482 <== osd.4 v2:172.21.15.124:6800/90863 1 ==== osd_op_reply(1 100000001f6.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) v8 ==== 164+0+0 (crc 0 0 0) 0x7f8e50009c10 con 0x55f8191da6a0 2023-10-06T01:42:11.989 INFO:teuthology.orchestra.run.smithi027.stderr:error getting xattr cephfs_data/100000001f6.00000000/symlink: (61) No data available 2023-10-06T01:42:11.989 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.982+0000 7f8e6e913000 1 -- 172.21.15.27:0/159494482 >> [v2:172.21.15.124:6800/90863,v1:172.21.15.124:6801/90863] conn(0x55f8191da6a0 msgr2=0x55f8191f0a70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:11.989 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.982+0000 7f8e6e913000 1 --2- 172.21.15.27:0/159494482 >> [v2:172.21.15.124:6800/90863,v1:172.21.15.124:6801/90863] conn(0x55f8191da6a0 0x55f8191f0a70 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:11.989 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.982+0000 7f8e6e913000 1 -- 172.21.15.27:0/159494482 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8e48065890 msgr2=0x7f8e48067d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:11.989 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.982+0000 7f8e6e913000 1 --2- 172.21.15.27:0/159494482 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f8e48065890 0x7f8e48067d40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8e60001160 tx=0x7f8e60016000 comp rx=0 tx=0).stop 2023-10-06T01:42:11.989 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.982+0000 7f8e6e913000 1 -- 172.21.15.27:0/159494482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e54002330 msgr2=0x55f8192775d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:11.989 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.982+0000 7f8e6e913000 1 --2- 172.21.15.27:0/159494482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f8e54002330 0x55f8192775d0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x55f8191dcbc0 tx=0x7f8e58007040 comp rx=0 tx=0).stop 2023-10-06T01:42:11.989 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.982+0000 7f8e6834f700 1 -- 172.21.15.27:0/159494482 reap_dead start 2023-10-06T01:42:11.989 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.982+0000 7f8e6e913000 1 -- 172.21.15.27:0/159494482 shutdown_connections 2023-10-06T01:42:11.989 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.982+0000 7f8e6e913000 1 -- 172.21.15.27:0/159494482 >> 172.21.15.27:0/159494482 conn(0x55f8191ad8a0 msgr2=0x55f8191adca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:11.989 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.982+0000 7f8e6e913000 1 -- 172.21.15.27:0/159494482 shutdown_connections 2023-10-06T01:42:11.989 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:11.982+0000 7f8e6e913000 1 -- 172.21.15.27:0/159494482 wait complete. 2023-10-06T01:42:11.990 ERROR:tasks.cephfs.filesystem:Command failed on smithi027 with status 1: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001f6.00000000 symlink' 2023-10-06T01:42:11.990 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mds mds_symlink_recovery true 2023-10-06T01:42:12.148 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.147+0000 7f47b5695700 1 Processor -- start 2023-10-06T01:42:12.148 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.147+0000 7f47b5695700 1 -- start start 2023-10-06T01:42:12.150 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.148+0000 7f47b5695700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f47b015dab0 0x7f47b00cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:12.150 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.148+0000 7f47b5695700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47b00b1110 con 0x7f47b01598d0 2023-10-06T01:42:12.150 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.148+0000 7f47b5695700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47b00b13b0 con 0x7f47b0159500 2023-10-06T01:42:12.150 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.148+0000 7f47b5695700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47b00b1690 con 0x7f47b015dab0 2023-10-06T01:42:12.150 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.148+0000 7f47ae7fc700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f47b01598d0 0x7f47b015d3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52158/0 (socket says 172.21.15.27:52158) 2023-10-06T01:42:12.150 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.148+0000 7f47ae7fc700 1 -- 172.21.15.27:0/1765924301 learned_addr learned my addr 172.21.15.27:0/1765924301 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:12.150 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.148+0000 7f47af7fe700 1 --2- 172.21.15.27:0/1765924301 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f47b015dab0 0x7f47b00cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:12.150 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.148+0000 7f47adffb700 1 -- 172.21.15.27:0/1765924301 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2241026356 0 0) 0x7f47b00b1110 con 0x7f47b01598d0 2023-10-06T01:42:12.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.148+0000 7f47adffb700 1 -- 172.21.15.27:0/1765924301 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4798003570 con 0x7f47b01598d0 2023-10-06T01:42:12.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.148+0000 7f47af7fe700 1 -- 172.21.15.27:0/1765924301 >> v1:172.21.15.124:6789/0 conn(0x7f47b0159500 legacy=0x7f47b00b3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:12.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.148+0000 7f47adffb700 1 -- 172.21.15.27:0/1765924301 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2997101281 0 0) 0x7f47b00b13b0 con 0x7f47b0159500 2023-10-06T01:42:12.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.148+0000 7f47af7fe700 1 -- 172.21.15.27:0/1765924301 >> v1:172.21.15.27:6789/0 conn(0x7f47b01598d0 legacy=0x7f47b015d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:12.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.148+0000 7f47af7fe700 1 -- 172.21.15.27:0/1765924301 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b00b1970 con 0x7f47b015dab0 2023-10-06T01:42:12.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.148+0000 7f47af7fe700 1 --2- 172.21.15.27:0/1765924301 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f47b015dab0 0x7f47b00cf6c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f47a8009e90 tx=0x7f47a8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72941296645220c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:12.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.148+0000 7f47adffb700 1 -- 172.21.15.27:0/1765924301 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a800e070 con 0x7f47b015dab0 2023-10-06T01:42:12.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.149+0000 7f47adffb700 1 -- 172.21.15.27:0/1765924301 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a800bc30 con 0x7f47b015dab0 2023-10-06T01:42:12.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.149+0000 7f47adffb700 1 -- 172.21.15.27:0/1765924301 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a8012430 con 0x7f47b015dab0 2023-10-06T01:42:12.154 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.149+0000 7f47b5695700 1 -- 172.21.15.27:0/1765924301 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f47b015dab0 msgr2=0x7f47b00cf6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:12.154 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.149+0000 7f47b5695700 1 --2- 172.21.15.27:0/1765924301 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f47b015dab0 0x7f47b00cf6c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f47a8009e90 tx=0x7f47a8009780 comp rx=0 tx=0).stop 2023-10-06T01:42:12.154 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.149+0000 7f47b5695700 1 -- 172.21.15.27:0/1765924301 shutdown_connections 2023-10-06T01:42:12.154 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.149+0000 7f47b5695700 1 --2- 172.21.15.27:0/1765924301 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f47b015dab0 0x7f47b00cf6c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:12.154 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.149+0000 7f47b5695700 1 -- 172.21.15.27:0/1765924301 >> 172.21.15.27:0/1765924301 conn(0x7f47b0067630 msgr2=0x7f47b0069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:12.154 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.149+0000 7f47b5695700 1 -- 172.21.15.27:0/1765924301 shutdown_connections 2023-10-06T01:42:12.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.149+0000 7f47b5695700 1 -- 172.21.15.27:0/1765924301 wait complete. 2023-10-06T01:42:12.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.150+0000 7f47b5695700 1 Processor -- start 2023-10-06T01:42:12.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.150+0000 7f47b5695700 1 -- start start 2023-10-06T01:42:12.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.151+0000 7f47b5695700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f47b0159500 0x7f47b014dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:12.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.151+0000 7f47b5695700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f47b01598d0 0x7f47b015b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:12.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.151+0000 7f47b5695700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47b015dab0 0x7f47b014bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:12.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.151+0000 7f47b5695700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b015c680 con 0x7f47b015dab0 2023-10-06T01:42:12.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.151+0000 7f47b5695700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b015ba70 con 0x7f47b01598d0 2023-10-06T01:42:12.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.151+0000 7f47b5695700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47b015bd50 con 0x7f47b0159500 2023-10-06T01:42:12.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.151+0000 7f47af7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47b015dab0 0x7f47b014bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:12.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.151+0000 7f47af7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47b015dab0 0x7f47b014bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37628/0 (socket says 172.21.15.27:37628) 2023-10-06T01:42:12.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.151+0000 7f47af7fe700 1 -- 172.21.15.27:0/2418458625 learned_addr learned my addr 172.21.15.27:0/2418458625 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:12.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.151+0000 7f47aeffd700 1 --2- 172.21.15.27:0/2418458625 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f47b0159500 0x7f47b014dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:12.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.152+0000 7f47ae7fc700 1 --2- 172.21.15.27:0/2418458625 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f47b01598d0 0x7f47b015b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:12.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.152+0000 7f47af7fe700 1 -- 172.21.15.27:0/2418458625 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f47b0159500 msgr2=0x7f47b014dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:12.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.152+0000 7f47af7fe700 1 --2- 172.21.15.27:0/2418458625 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f47b0159500 0x7f47b014dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:12.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.152+0000 7f47af7fe700 1 -- 172.21.15.27:0/2418458625 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f47b01598d0 msgr2=0x7f47b015b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:12.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.152+0000 7f47af7fe700 1 --2- 172.21.15.27:0/2418458625 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f47b01598d0 0x7f47b015b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:12.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.152+0000 7f47af7fe700 1 -- 172.21.15.27:0/2418458625 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a8009a20 con 0x7f47b015dab0 2023-10-06T01:42:12.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.152+0000 7f47ae7fc700 1 --2- 172.21.15.27:0/2418458625 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f47b01598d0 0x7f47b015b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:42:12.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.152+0000 7f47af7fe700 1 --2- 172.21.15.27:0/2418458625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47b015dab0 0x7f47b014bf60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f47a8013000 tx=0x7f47a8019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:12.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.152+0000 7f4796ffd700 1 -- 172.21.15.27:0/2418458625 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a801b630 con 0x7f47b015dab0 2023-10-06T01:42:12.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.152+0000 7f4796ffd700 1 -- 172.21.15.27:0/2418458625 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a801b7b0 con 0x7f47b015dab0 2023-10-06T01:42:12.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.152+0000 7f47b5695700 1 -- 172.21.15.27:0/2418458625 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47b015c030 con 0x7f47b015dab0 2023-10-06T01:42:12.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.152+0000 7f4796ffd700 1 -- 172.21.15.27:0/2418458625 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a801bb70 con 0x7f47b015dab0 2023-10-06T01:42:12.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.152+0000 7f47b5695700 1 -- 172.21.15.27:0/2418458625 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47b015b780 con 0x7f47b015dab0 2023-10-06T01:42:12.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.153+0000 7f47b5695700 1 -- 172.21.15.27:0/2418458625 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47b0064050 con 0x7f47b015dab0 2023-10-06T01:42:12.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.153+0000 7f4796ffd700 1 -- 172.21.15.27:0/2418458625 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f47a801c400 con 0x7f47b015dab0 2023-10-06T01:42:12.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.154+0000 7f4796ffd700 1 --2- 172.21.15.27:0/2418458625 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f479c066f30 0x7f479c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:12.158 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.154+0000 7f4796ffd700 1 -- 172.21.15.27:0/2418458625 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f47a80864f0 con 0x7f47b015dab0 2023-10-06T01:42:12.158 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.154+0000 7f47aeffd700 1 --2- 172.21.15.27:0/2418458625 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f479c066f30 0x7f479c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:12.158 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.154+0000 7f47aeffd700 1 --2- 172.21.15.27:0/2418458625 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f479c066f30 0x7f479c0693e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f47a00011e0 tx=0x7f47a0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:12.159 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.158+0000 7f4796ffd700 1 -- 172.21.15.27:0/2418458625 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f47a8053600 con 0x7f47b015dab0 2023-10-06T01:42:12.353 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.352+0000 7f47b5695700 1 -- 172.21.15.27:0/2418458625 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mds_symlink_recovery}] v 0) v1 -- 0x7f47b00d0420 con 0x7f47b015dab0 2023-10-06T01:42:12.356 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.355+0000 7f4796ffd700 1 -- 172.21.15.27:0/2418458625 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mds_symlink_recovery}]=0 v12)=0 v12) v1 ==== 121+0+0 (secure 0 0 0) 0x7f47a8056c20 con 0x7f47b015dab0 2023-10-06T01:42:12.359 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.357+0000 7f47b5695700 1 -- 172.21.15.27:0/2418458625 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f479c066f30 msgr2=0x7f479c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:12.359 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.357+0000 7f47b5695700 1 --2- 172.21.15.27:0/2418458625 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f479c066f30 0x7f479c0693e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f47a00011e0 tx=0x7f47a0009040 comp rx=0 tx=0).stop 2023-10-06T01:42:12.359 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.357+0000 7f47b5695700 1 -- 172.21.15.27:0/2418458625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47b015dab0 msgr2=0x7f47b014bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:12.359 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.357+0000 7f47b5695700 1 --2- 172.21.15.27:0/2418458625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47b015dab0 0x7f47b014bf60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f47a8013000 tx=0x7f47a8019000 comp rx=0 tx=0).stop 2023-10-06T01:42:12.359 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.357+0000 7f47b5695700 1 -- 172.21.15.27:0/2418458625 shutdown_connections 2023-10-06T01:42:12.359 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.357+0000 7f47b5695700 1 --2- 172.21.15.27:0/2418458625 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f479c066f30 0x7f479c0693e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:12.359 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.357+0000 7f47b5695700 1 --2- 172.21.15.27:0/2418458625 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f47b0159500 0x7f47b014dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:12.360 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.357+0000 7f47b5695700 1 --2- 172.21.15.27:0/2418458625 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f47b015dab0 0x7f47b014bf60 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:12.360 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.357+0000 7f47b5695700 1 --2- 172.21.15.27:0/2418458625 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f47b01598d0 0x7f47b015b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:12.360 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.357+0000 7f47b5695700 1 -- 172.21.15.27:0/2418458625 >> 172.21.15.27:0/2418458625 conn(0x7f47b0067630 msgr2=0x7f47b00aff40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:12.360 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.358+0000 7f47b5695700 1 -- 172.21.15.27:0/2418458625 shutdown_connections 2023-10-06T01:42:12.360 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.358+0000 7f47b5695700 1 -- 172.21.15.27:0/2418458625 wait complete. 2023-10-06T01:42:12.372 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T01:42:12.372 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir sym_dir) 2023-10-06T01:42:12.389 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T01:42:12.389 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch sym_dir/file1) 2023-10-06T01:42:12.447 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T01:42:12.447 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln -s ./file1 sym_dir/symlink_file1) 2023-10-06T01:42:12.503 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits daemon-helper kill python3 -c ' 2023-10-06T01:42:12.503 DEBUG:teuthology.orchestra.run.smithi027:> import os 2023-10-06T01:42:12.503 DEBUG:teuthology.orchestra.run.smithi027:> import stat 2023-10-06T01:42:12.503 DEBUG:teuthology.orchestra.run.smithi027:> 2023-10-06T01:42:12.503 DEBUG:teuthology.orchestra.run.smithi027:> print(os.lstat("/home/ubuntu/cephtest/mnt.0/sym_dir/symlink_file1").st_ino) 2023-10-06T01:42:12.503 DEBUG:teuthology.orchestra.run.smithi027:> ' 2023-10-06T01:42:12.608 INFO:teuthology.orchestra.run.smithi027.stdout:1099511628281 2023-10-06T01:42:12.797 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T01:42:12.956 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.955+0000 7ff372c29700 1 Processor -- start 2023-10-06T01:42:12.956 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.955+0000 7ff372c29700 1 -- start start 2023-10-06T01:42:12.959 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.956+0000 7ff372c29700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff36c153f70 0x7ff36c0b0f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:12.959 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.956+0000 7ff372c29700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff36c0ad4d0 0x7ff36c0ae0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:12.959 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.956+0000 7ff372c29700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff36c15a5e0 con 0x7ff36c153ba0 2023-10-06T01:42:12.959 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.956+0000 7ff372c29700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff36c15a8c0 con 0x7ff36c0ad4d0 2023-10-06T01:42:12.959 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.956+0000 7ff372c29700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff36c15aba0 con 0x7ff36c153f70 2023-10-06T01:42:12.959 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.956+0000 7ff372428700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff36c0ad4d0 0x7ff36c0ae0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:12.959 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.956+0000 7ff372428700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff36c0ad4d0 0x7ff36c0ae0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37642/0 (socket says 172.21.15.27:37642) 2023-10-06T01:42:12.959 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.956+0000 7ff372428700 1 -- 172.21.15.27:0/3841161158 learned_addr learned my addr 172.21.15.27:0/3841161158 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:12.959 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.956+0000 7ff371426700 1 --2- 172.21.15.27:0/3841161158 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff36c153f70 0x7ff36c0b0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:12.960 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.956+0000 7ff372428700 1 -- 172.21.15.27:0/3841161158 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff36c153f70 msgr2=0x7ff36c0b0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:12.960 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.956+0000 7ff372428700 1 --2- 172.21.15.27:0/3841161158 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff36c153f70 0x7ff36c0b0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:12.960 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.956+0000 7ff372428700 1 -- 172.21.15.27:0/3841161158 >> v1:172.21.15.124:6789/0 conn(0x7ff36c153ba0 legacy=0x7ff36c159d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:12.960 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.956+0000 7ff372428700 1 -- 172.21.15.27:0/3841161158 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff36c0accd0 con 0x7ff36c0ad4d0 2023-10-06T01:42:12.960 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.957+0000 7ff372428700 1 --2- 172.21.15.27:0/3841161158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff36c0ad4d0 0x7ff36c0ae0c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7ff35c009f20 tx=0x7ff35c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9c4cb4646eb5628 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:12.960 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.957+0000 7ff370c25700 1 -- 172.21.15.27:0/3841161158 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff35c00e070 con 0x7ff36c0ad4d0 2023-10-06T01:42:12.960 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.957+0000 7ff370c25700 1 -- 172.21.15.27:0/3841161158 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff35c00b9d0 con 0x7ff36c0ad4d0 2023-10-06T01:42:12.960 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.957+0000 7ff370c25700 1 -- 172.21.15.27:0/3841161158 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff35c0092e0 con 0x7ff36c0ad4d0 2023-10-06T01:42:12.960 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.957+0000 7ff372c29700 1 -- 172.21.15.27:0/3841161158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff36c0ad4d0 msgr2=0x7ff36c0ae0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:12.960 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.957+0000 7ff372c29700 1 --2- 172.21.15.27:0/3841161158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff36c0ad4d0 0x7ff36c0ae0c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7ff35c009f20 tx=0x7ff35c009840 comp rx=0 tx=0).stop 2023-10-06T01:42:12.960 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.957+0000 7ff372c29700 1 -- 172.21.15.27:0/3841161158 shutdown_connections 2023-10-06T01:42:12.961 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.957+0000 7ff372c29700 1 --2- 172.21.15.27:0/3841161158 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff36c0ad4d0 0x7ff36c0ae0c0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:12.961 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.957+0000 7ff372c29700 1 --2- 172.21.15.27:0/3841161158 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff36c153f70 0x7ff36c0b0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:12.961 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.957+0000 7ff372c29700 1 -- 172.21.15.27:0/3841161158 >> 172.21.15.27:0/3841161158 conn(0x7ff36c06e600 msgr2=0x7ff36c06afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:12.961 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.958+0000 7ff372c29700 1 -- 172.21.15.27:0/3841161158 shutdown_connections 2023-10-06T01:42:12.961 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.958+0000 7ff372c29700 1 -- 172.21.15.27:0/3841161158 wait complete. 2023-10-06T01:42:12.963 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.959+0000 7ff372c29700 1 Processor -- start 2023-10-06T01:42:12.963 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.959+0000 7ff372c29700 1 -- start start 2023-10-06T01:42:12.964 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.959+0000 7ff372c29700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff36c0ad4d0 0x7ff36c0abea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:12.964 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.959+0000 7ff372c29700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff36c153ba0 0x7ff36c14a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:12.964 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.959+0000 7ff372c29700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff36c153f70 0x7ff36c0aa3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:12.964 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.959+0000 7ff372c29700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff36c069d30 con 0x7ff36c153ba0 2023-10-06T01:42:12.964 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.959+0000 7ff372c29700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff36c0ac7f0 con 0x7ff36c153f70 2023-10-06T01:42:12.964 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.959+0000 7ff372c29700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff36c06a530 con 0x7ff36c0ad4d0 2023-10-06T01:42:12.964 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.960+0000 7ff371426700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff36c153ba0 0x7ff36c14a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:12.964 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.960+0000 7ff371426700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff36c153ba0 0x7ff36c14a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37652/0 (socket says 172.21.15.27:37652) 2023-10-06T01:42:12.964 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.960+0000 7ff371426700 1 -- 172.21.15.27:0/2896513824 learned_addr learned my addr 172.21.15.27:0/2896513824 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:12.964 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.960+0000 7ff372428700 1 --2- 172.21.15.27:0/2896513824 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff36c153f70 0x7ff36c0aa3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:12.965 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.960+0000 7ff371c27700 1 --2- 172.21.15.27:0/2896513824 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff36c0ad4d0 0x7ff36c0abea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:12.965 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.960+0000 7ff371426700 1 -- 172.21.15.27:0/2896513824 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff36c0ad4d0 msgr2=0x7ff36c0abea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:12.965 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.960+0000 7ff371426700 1 --2- 172.21.15.27:0/2896513824 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff36c0ad4d0 0x7ff36c0abea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:12.965 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.960+0000 7ff371426700 1 -- 172.21.15.27:0/2896513824 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff36c153f70 msgr2=0x7ff36c0aa3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:12.965 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.960+0000 7ff371426700 1 --2- 172.21.15.27:0/2896513824 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff36c153f70 0x7ff36c0aa3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:12.965 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.960+0000 7ff371426700 1 -- 172.21.15.27:0/2896513824 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff35c009ae0 con 0x7ff36c153ba0 2023-10-06T01:42:12.965 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.960+0000 7ff371426700 1 --2- 172.21.15.27:0/2896513824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff36c153ba0 0x7ff36c14a060 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7ff36000ec80 tx=0x7ff3600091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:12.965 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.960+0000 7ff369ffb700 1 -- 172.21.15.27:0/2896513824 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff36000c850 con 0x7ff36c153ba0 2023-10-06T01:42:12.965 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.960+0000 7ff369ffb700 1 -- 172.21.15.27:0/2896513824 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff36000c9b0 con 0x7ff36c153ba0 2023-10-06T01:42:12.965 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.960+0000 7ff372c29700 1 -- 172.21.15.27:0/2896513824 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff36c069f20 con 0x7ff36c153ba0 2023-10-06T01:42:12.965 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.960+0000 7ff369ffb700 1 -- 172.21.15.27:0/2896513824 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff36000cd70 con 0x7ff36c153ba0 2023-10-06T01:42:12.966 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.960+0000 7ff372c29700 1 -- 172.21.15.27:0/2896513824 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff36c14a5a0 con 0x7ff36c153ba0 2023-10-06T01:42:12.966 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.961+0000 7ff372c29700 1 -- 172.21.15.27:0/2896513824 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff36c06e1d0 con 0x7ff36c153ba0 2023-10-06T01:42:12.966 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.962+0000 7ff369ffb700 1 -- 172.21.15.27:0/2896513824 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff360013680 con 0x7ff36c153ba0 2023-10-06T01:42:12.966 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.962+0000 7ff369ffb700 1 --2- 172.21.15.27:0/2896513824 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7ff358066f30 0x7ff3580693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:12.966 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.962+0000 7ff369ffb700 1 -- 172.21.15.27:0/2896513824 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6058+0+0 (secure 0 0 0) 0x7ff360084880 con 0x7ff36c153ba0 2023-10-06T01:42:12.966 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.962+0000 7ff371c27700 1 --2- 172.21.15.27:0/2896513824 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7ff358066f30 0x7ff3580693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:12.966 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.963+0000 7ff371c27700 1 --2- 172.21.15.27:0/2896513824 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7ff358066f30 0x7ff3580693e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff364007fd0 tx=0x7ff364007eb0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:12.968 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:12.966+0000 7ff369ffb700 1 -- 172.21.15.27:0/2896513824 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff360051990 con 0x7ff36c153ba0 2023-10-06T01:42:13.205 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.204+0000 7ff372c29700 1 -- 172.21.15.27:0/2896513824 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff36c04c3c0 con 0x7ff36c153ba0 2023-10-06T01:42:13.206 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.204+0000 7ff369ffb700 1 -- 172.21.15.27:0/2896513824 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4597 (secure 0 0 0) 0x7ff360054fb0 con 0x7ff36c153ba0 2023-10-06T01:42:13.206 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:13.206 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4870,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6836/1043044736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":1043044736},{"type":"v1","addr":"172.21.15.124:6836","nonce":1043044736}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4962,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/1207987117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1207987117},{"type":"v1","addr":"172.21.15.27:6834","nonce":1207987117}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4965,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/943261840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":943261840},{"type":"v1","addr":"172.21.15.27:6835","nonce":943261840}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T01:42:03.036784+0000","modified":"2023-10-06T01:42:06.652163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4974},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4974":{"gid":4974,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.124:6837/325247112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6835","nonce":325247112},{"type":"v1","addr":"172.21.15.124:6837","nonce":325247112}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-10-06T01:42:13.208 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.207+0000 7ff372c29700 1 -- 172.21.15.27:0/2896513824 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7ff358066f30 msgr2=0x7ff3580693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:13.208 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.207+0000 7ff372c29700 1 --2- 172.21.15.27:0/2896513824 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7ff358066f30 0x7ff3580693e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ff364007fd0 tx=0x7ff364007eb0 comp rx=0 tx=0).stop 2023-10-06T01:42:13.208 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.207+0000 7ff372c29700 1 -- 172.21.15.27:0/2896513824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff36c153ba0 msgr2=0x7ff36c14a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:13.208 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.207+0000 7ff372c29700 1 --2- 172.21.15.27:0/2896513824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff36c153ba0 0x7ff36c14a060 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7ff36000ec80 tx=0x7ff3600091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:13.208 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.207+0000 7ff372c29700 1 -- 172.21.15.27:0/2896513824 shutdown_connections 2023-10-06T01:42:13.209 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.207+0000 7ff372c29700 1 --2- 172.21.15.27:0/2896513824 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7ff358066f30 0x7ff3580693e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:13.209 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.207+0000 7ff372c29700 1 --2- 172.21.15.27:0/2896513824 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff36c0ad4d0 0x7ff36c0abea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:13.209 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.207+0000 7ff372c29700 1 --2- 172.21.15.27:0/2896513824 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff36c153f70 0x7ff36c0aa3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:13.209 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.207+0000 7ff372c29700 1 --2- 172.21.15.27:0/2896513824 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff36c153ba0 0x7ff36c14a060 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:13.209 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.207+0000 7ff372c29700 1 -- 172.21.15.27:0/2896513824 >> 172.21.15.27:0/2896513824 conn(0x7ff36c06e600 msgr2=0x7ff36c15bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:13.209 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.207+0000 7ff372c29700 1 -- 172.21.15.27:0/2896513824 shutdown_connections 2023-10-06T01:42:13.209 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.207+0000 7ff372c29700 1 -- 172.21.15.27:0/2896513824 wait complete. 2023-10-06T01:42:13.209 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 15 2023-10-06T01:42:13.220 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json flush journal 2023-10-06T01:42:13.367 INFO:teuthology.orchestra.run.smithi124.stdout:{"message":"","return_code":0} 2023-10-06T01:42:13.367 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2023-10-06T01:42:13.368 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001f9.00000000 symlink 2023-10-06T01:42:13.407 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.406+0000 7f93a9299000 1 Processor -- start 2023-10-06T01:42:13.407 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.406+0000 7f93a9299000 1 -- start start 2023-10-06T01:42:13.409 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.407+0000 7f93a9299000 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x562f0c924cb0 0x562f0c9ff8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:13.410 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.407+0000 7f93a9299000 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x562f0c94cde0 0x562f0c951990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:13.410 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.407+0000 7f93a9299000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f0c9f7290 0x562f0c9500c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:13.410 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.407+0000 7f93a9299000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f0c86d980 con 0x562f0c9f7290 2023-10-06T01:42:13.410 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.407+0000 7f93a9299000 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f0c86d620 con 0x562f0c94cde0 2023-10-06T01:42:13.410 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.407+0000 7f93a9299000 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x562f0c864400 con 0x562f0c924cb0 2023-10-06T01:42:13.410 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.407+0000 7f93a2cd5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f0c9f7290 0x562f0c9500c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:13.410 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.407+0000 7f93a2cd5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f0c9f7290 0x562f0c9500c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37676/0 (socket says 172.21.15.27:37676) 2023-10-06T01:42:13.410 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.407+0000 7f93a2cd5700 1 -- 172.21.15.27:0/1111301306 learned_addr learned my addr 172.21.15.27:0/1111301306 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:13.410 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.407+0000 7f93a24d4700 1 --2- 172.21.15.27:0/1111301306 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x562f0c924cb0 0x562f0c9ff8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:13.410 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.407+0000 7f93a1cd3700 1 --2- 172.21.15.27:0/1111301306 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x562f0c94cde0 0x562f0c951990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:13.411 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.407+0000 7f93a2cd5700 1 -- 172.21.15.27:0/1111301306 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x562f0c924cb0 msgr2=0x562f0c9ff8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:13.411 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.407+0000 7f93a2cd5700 1 --2- 172.21.15.27:0/1111301306 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x562f0c924cb0 0x562f0c9ff8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:13.411 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.407+0000 7f93a2cd5700 1 -- 172.21.15.27:0/1111301306 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x562f0c94cde0 msgr2=0x562f0c951990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:13.411 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.407+0000 7f93a2cd5700 1 --2- 172.21.15.27:0/1111301306 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x562f0c94cde0 0x562f0c951990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:13.411 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.407+0000 7f93a2cd5700 1 -- 172.21.15.27:0/1111301306 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f939c00a180 con 0x562f0c9f7290 2023-10-06T01:42:13.411 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.407+0000 7f93a2cd5700 1 --2- 172.21.15.27:0/1111301306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f0c9f7290 0x562f0c9500c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f939c00e5f0 tx=0x7f939c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=564e311ef5be6f16 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:13.411 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.407+0000 7f93a0cd1700 1 -- 172.21.15.27:0/1111301306 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939c00ee70 con 0x562f0c9f7290 2023-10-06T01:42:13.411 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.407+0000 7f93a0cd1700 1 -- 172.21.15.27:0/1111301306 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f939c0057e0 con 0x562f0c9f7290 2023-10-06T01:42:13.411 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.408+0000 7f93a0cd1700 1 -- 172.21.15.27:0/1111301306 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939c005950 con 0x562f0c9f7290 2023-10-06T01:42:13.411 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.408+0000 7f93a9299000 1 -- 172.21.15.27:0/1111301306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f0c9f7290 msgr2=0x562f0c9500c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:13.411 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.408+0000 7f93a9299000 1 --2- 172.21.15.27:0/1111301306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f0c9f7290 0x562f0c9500c0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f939c00e5f0 tx=0x7f939c014000 comp rx=0 tx=0).stop 2023-10-06T01:42:13.412 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.408+0000 7f93a9299000 1 -- 172.21.15.27:0/1111301306 shutdown_connections 2023-10-06T01:42:13.412 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.408+0000 7f93a9299000 1 --2- 172.21.15.27:0/1111301306 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x562f0c924cb0 0x562f0c9ff8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:13.412 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.408+0000 7f93a9299000 1 --2- 172.21.15.27:0/1111301306 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f0c9f7290 0x562f0c9500c0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:13.412 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.408+0000 7f93a9299000 1 --2- 172.21.15.27:0/1111301306 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x562f0c94cde0 0x562f0c951990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:13.412 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.408+0000 7f93a9299000 1 -- 172.21.15.27:0/1111301306 >> 172.21.15.27:0/1111301306 conn(0x562f0c9238a0 msgr2=0x562f0c9f37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:13.412 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.408+0000 7f93a9299000 1 -- 172.21.15.27:0/1111301306 shutdown_connections 2023-10-06T01:42:13.412 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.408+0000 7f93a9299000 1 -- 172.21.15.27:0/1111301306 wait complete. 2023-10-06T01:42:13.412 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.409+0000 7f93a9299000 1 Processor -- start 2023-10-06T01:42:13.412 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.409+0000 7f93a9299000 1 -- start start 2023-10-06T01:42:13.415 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.409+0000 7f93a9299000 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x562f0c924cb0 0x562f0c9ed410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:13.415 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.409+0000 7f93a9299000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f0c94cde0 0x562f0c94fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:13.415 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.409+0000 7f93a9299000 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x562f0c9f7290 0x562f0c969520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:13.415 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.409+0000 7f93a9299000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f0c969c30 con 0x562f0c94cde0 2023-10-06T01:42:13.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.409+0000 7f93a9299000 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f0c951200 con 0x562f0c924cb0 2023-10-06T01:42:13.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.409+0000 7f93a9299000 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x562f0c9514a0 con 0x562f0c9f7290 2023-10-06T01:42:13.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.409+0000 7f93a1cd3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f0c94cde0 0x562f0c94fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:13.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.409+0000 7f93a1cd3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f0c94cde0 0x562f0c94fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37680/0 (socket says 172.21.15.27:37680) 2023-10-06T01:42:13.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.409+0000 7f93a1cd3700 1 -- 172.21.15.27:0/3335089406 learned_addr learned my addr 172.21.15.27:0/3335089406 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:13.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.409+0000 7f93a2cd5700 1 --2- 172.21.15.27:0/3335089406 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x562f0c9f7290 0x562f0c969520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:13.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.409+0000 7f93a24d4700 1 --2- 172.21.15.27:0/3335089406 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x562f0c924cb0 0x562f0c9ed410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:13.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.410+0000 7f93a1cd3700 1 -- 172.21.15.27:0/3335089406 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x562f0c9f7290 msgr2=0x562f0c969520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:13.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.410+0000 7f93a1cd3700 1 --2- 172.21.15.27:0/3335089406 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x562f0c9f7290 0x562f0c969520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:13.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.410+0000 7f93a1cd3700 1 -- 172.21.15.27:0/3335089406 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x562f0c924cb0 msgr2=0x562f0c9ed410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:13.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.410+0000 7f93a1cd3700 1 --2- 172.21.15.27:0/3335089406 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x562f0c924cb0 0x562f0c9ed410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:13.416 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.410+0000 7f93a1cd3700 1 -- 172.21.15.27:0/3335089406 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562f0c8961c0 con 0x562f0c94cde0 2023-10-06T01:42:13.417 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.410+0000 7f93a1cd3700 1 --2- 172.21.15.27:0/3335089406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f0c94cde0 0x562f0c94fbb0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f938c00ab10 tx=0x7f938c00e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:13.417 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.410+0000 7f939affd700 1 -- 172.21.15.27:0/3335089406 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938c011040 con 0x562f0c94cde0 2023-10-06T01:42:13.417 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.410+0000 7f939affd700 1 -- 172.21.15.27:0/3335089406 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f938c008ac0 con 0x562f0c94cde0 2023-10-06T01:42:13.417 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.410+0000 7f93a9299000 1 -- 172.21.15.27:0/3335089406 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x562f0c87c3d0 con 0x562f0c94cde0 2023-10-06T01:42:13.417 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.410+0000 7f939affd700 1 -- 172.21.15.27:0/3335089406 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938c008e40 con 0x562f0c94cde0 2023-10-06T01:42:13.417 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.410+0000 7f93a9299000 1 -- 172.21.15.27:0/3335089406 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562f0c86fae0 con 0x562f0c94cde0 2023-10-06T01:42:13.417 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.411+0000 7f939affd700 1 -- 172.21.15.27:0/3335089406 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f938c016690 con 0x562f0c94cde0 2023-10-06T01:42:13.417 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.412+0000 7f939affd700 1 --2- 172.21.15.27:0/3335089406 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9384065890 0x7f9384067d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:13.417 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.412+0000 7f939affd700 1 -- 172.21.15.27:0/3335089406 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f938c083550 con 0x562f0c94cde0 2023-10-06T01:42:13.417 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.412+0000 7f93a24d4700 1 --2- 172.21.15.27:0/3335089406 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9384065890 0x7f9384067d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:13.417 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.412+0000 7f93a9299000 1 --2- 172.21.15.27:0/3335089406 >> [v2:172.21.15.124:6816/90885,v1:172.21.15.124:6817/90885] conn(0x562f0ca09db0 0x562f0ca0a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:13.417 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.412+0000 7f93a9299000 1 -- 172.21.15.27:0/3335089406 --> [v2:172.21.15.124:6816/90885,v1:172.21.15.124:6817/90885] -- osd_op(unknown.0.0:1 6.36 6:6ee5e2c4:::100000001f9.00000000:head [getxattr symlink in=7b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) v8 -- 0x562f0ca0a6c0 con 0x562f0ca09db0 2023-10-06T01:42:13.418 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.412+0000 7f93a2cd5700 1 --2- 172.21.15.27:0/3335089406 >> [v2:172.21.15.124:6816/90885,v1:172.21.15.124:6817/90885] conn(0x562f0ca09db0 0x562f0ca0a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:13.418 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.412+0000 7f93a24d4700 1 --2- 172.21.15.27:0/3335089406 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9384065890 0x7f9384067d40 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9394000c00 tx=0x7f9394001780 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:13.418 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.412+0000 7f93a2cd5700 1 --2- 172.21.15.27:0/3335089406 >> [v2:172.21.15.124:6816/90885,v1:172.21.15.124:6817/90885] conn(0x562f0ca09db0 0x562f0ca0a180 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:13.418 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.413+0000 7f93a2cd5700 1 -- 172.21.15.27:0/3335089406 <== osd.5 v2:172.21.15.124:6816/90885 1 ==== osd_op_reply(1 100000001f9.00000000 [getxattr (11) out=11b] v0'0 uv1 ondisk = 0) v8 ==== 164+0+11 (crc 0 0 0) 0x7f939c00e7e0 con 0x562f0ca09db0 2023-10-06T01:42:13.418 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.413+0000 7f93a9299000 1 -- 172.21.15.27:0/3335089406 >> [v2:172.21.15.124:6816/90885,v1:172.21.15.124:6817/90885] conn(0x562f0ca09db0 msgr2=0x562f0ca0a180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:13.418 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.413+0000 7f93a9299000 1 --2- 172.21.15.27:0/3335089406 >> [v2:172.21.15.124:6816/90885,v1:172.21.15.124:6817/90885] conn(0x562f0ca09db0 0x562f0ca0a180 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:13.418 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.413+0000 7f93a9299000 1 -- 172.21.15.27:0/3335089406 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9384065890 msgr2=0x7f9384067d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:13.418 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.413+0000 7f93a9299000 1 --2- 172.21.15.27:0/3335089406 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9384065890 0x7f9384067d40 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9394000c00 tx=0x7f9394001780 comp rx=0 tx=0).stop 2023-10-06T01:42:13.418 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.413+0000 7f93a9299000 1 -- 172.21.15.27:0/3335089406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f0c94cde0 msgr2=0x562f0c94fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:13.418 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.413+0000 7f93a9299000 1 --2- 172.21.15.27:0/3335089406 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x562f0c94cde0 0x562f0c94fbb0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f938c00ab10 tx=0x7f938c00e040 comp rx=0 tx=0).stop 2023-10-06T01:42:13.418 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.413+0000 7f93a2cd5700 1 -- 172.21.15.27:0/3335089406 reap_dead start 2023-10-06T01:42:13.418 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.413+0000 7f93a9299000 1 -- 172.21.15.27:0/3335089406 shutdown_connections 2023-10-06T01:42:13.419 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.413+0000 7f93a9299000 1 -- 172.21.15.27:0/3335089406 >> 172.21.15.27:0/3335089406 conn(0x562f0c9238a0 msgr2=0x562f0c9ff400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:13.419 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.413+0000 7f93a9299000 1 -- 172.21.15.27:0/3335089406 shutdown_connections 2023-10-06T01:42:13.419 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:13.413+0000 7f93a9299000 1 -- 172.21.15.27:0/3335089406 wait complete. 2023-10-06T01:42:13.419 INFO:teuthology.orchestra.run.smithi027.stdout:./file1 2023-10-06T01:42:13.420 DEBUG:teuthology.orchestra.run.smithi027:> ceph-dencoder type string_wrapper import - decode dump_json 2023-10-06T01:42:13.585 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2023-10-06T01:42:13.585 INFO:teuthology.orchestra.run.smithi027.stdout: "s": "./file1" 2023-10-06T01:42:13.585 INFO:teuthology.orchestra.run.smithi027.stdout:} 2023-10-06T01:42:13.585 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:13.587 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T01:42:13.587 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parent_a) 2023-10-06T01:42:13.603 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T01:42:13.604 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch parent_a/alpha) 2023-10-06T01:42:13.662 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits daemon-helper kill python3 -c ' 2023-10-06T01:42:13.662 DEBUG:teuthology.orchestra.run.smithi027:> import os 2023-10-06T01:42:13.662 DEBUG:teuthology.orchestra.run.smithi027:> import stat 2023-10-06T01:42:13.662 DEBUG:teuthology.orchestra.run.smithi027:> 2023-10-06T01:42:13.662 DEBUG:teuthology.orchestra.run.smithi027:> print(os.stat("/home/ubuntu/cephtest/mnt.0/parent_a/alpha").st_ino) 2023-10-06T01:42:13.662 DEBUG:teuthology.orchestra.run.smithi027:> ' 2023-10-06T01:42:13.767 INFO:teuthology.orchestra.run.smithi027.stdout:1099511628283 2023-10-06T01:42:13.956 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T01:42:14.116 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.115+0000 7fafa5528700 1 Processor -- start 2023-10-06T01:42:14.116 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.115+0000 7fafa5528700 1 -- start start 2023-10-06T01:42:14.119 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.116+0000 7fafa5528700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fafa00cdc00 0x7fafa00cbe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:14.119 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.116+0000 7fafa5528700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fafa00d1030 0x7fafa00c66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:14.119 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.116+0000 7fafa5528700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa00ae140 con 0x7fafa00c6bf0 2023-10-06T01:42:14.119 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.116+0000 7fafa5528700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafa00cc390 con 0x7fafa00cdc00 2023-10-06T01:42:14.119 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.116+0000 7fafa5528700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafa00c7070 con 0x7fafa00d1030 2023-10-06T01:42:14.119 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.116+0000 7faf9f7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fafa00c6bf0 0x7fafa00ae5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52170/0 (socket says 172.21.15.27:52170) 2023-10-06T01:42:14.121 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.116+0000 7faf9f7fe700 1 -- 172.21.15.27:0/987964446 learned_addr learned my addr 172.21.15.27:0/987964446 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:14.121 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.116+0000 7faf9e7fc700 1 --2- 172.21.15.27:0/987964446 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fafa00d1030 0x7fafa00c66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:14.121 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.116+0000 7faf9effd700 1 --2- 172.21.15.27:0/987964446 >> v2:172.21.15.124:3300/0 conn(0x7fafa00cdc00 0x7fafa00cbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:14.121 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.116+0000 7faf9dffb700 1 -- 172.21.15.27:0/987964446 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1569887622 0 0) 0x7fafa00ae140 con 0x7fafa00c6bf0 2023-10-06T01:42:14.121 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.116+0000 7faf9dffb700 1 -- 172.21.15.27:0/987964446 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf8c003570 con 0x7fafa00c6bf0 2023-10-06T01:42:14.121 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.116+0000 7faf9e7fc700 1 -- 172.21.15.27:0/987964446 >> v2:172.21.15.124:3300/0 conn(0x7fafa00cdc00 msgr2=0x7fafa00cbe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:14.121 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.116+0000 7faf9dffb700 1 -- 172.21.15.27:0/987964446 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4005461938 0 0) 0x7faf8c003570 con 0x7fafa00c6bf0 2023-10-06T01:42:14.121 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.116+0000 7faf9e7fc700 1 --2- 172.21.15.27:0/987964446 >> v2:172.21.15.124:3300/0 conn(0x7fafa00cdc00 0x7fafa00cbe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:14.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.116+0000 7faf9e7fc700 1 -- 172.21.15.27:0/987964446 >> v1:172.21.15.27:6789/0 conn(0x7fafa00c6bf0 legacy=0x7fafa00ae5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:14.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.116+0000 7faf9e7fc700 1 -- 172.21.15.27:0/987964446 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa00ad530 con 0x7fafa00d1030 2023-10-06T01:42:14.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.116+0000 7faf9effd700 1 --2- 172.21.15.27:0/987964446 >> v2:172.21.15.124:3300/0 conn(0x7fafa00cdc00 0x7fafa00cbe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:42:14.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.117+0000 7faf9e7fc700 1 --2- 172.21.15.27:0/987964446 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fafa00d1030 0x7fafa00c66b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7faf880098e0 tx=0x7faf880091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c13606c3ff179755 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:14.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.117+0000 7faf9dffb700 1 -- 172.21.15.27:0/987964446 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf98001470 con 0x7fafa00d1030 2023-10-06T01:42:14.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.117+0000 7faf9dffb700 1 -- 172.21.15.27:0/987964446 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf88011420 con 0x7fafa00d1030 2023-10-06T01:42:14.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.117+0000 7faf9dffb700 1 -- 172.21.15.27:0/987964446 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf880117c0 con 0x7fafa00d1030 2023-10-06T01:42:14.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.117+0000 7fafa5528700 1 -- 172.21.15.27:0/987964446 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fafa00d1030 msgr2=0x7fafa00c66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:14.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.117+0000 7fafa5528700 1 --2- 172.21.15.27:0/987964446 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fafa00d1030 0x7fafa00c66b0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7faf880098e0 tx=0x7faf880091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:14.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.117+0000 7fafa5528700 1 -- 172.21.15.27:0/987964446 shutdown_connections 2023-10-06T01:42:14.122 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.117+0000 7fafa5528700 1 --2- 172.21.15.27:0/987964446 >> v2:172.21.15.124:3300/0 conn(0x7fafa00cdc00 0x7fafa00cbe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:14.123 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.117+0000 7fafa5528700 1 --2- 172.21.15.27:0/987964446 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fafa00d1030 0x7fafa00c66b0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:14.123 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.117+0000 7fafa5528700 1 -- 172.21.15.27:0/987964446 >> 172.21.15.27:0/987964446 conn(0x7fafa0099990 msgr2=0x7fafa00b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:14.123 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.117+0000 7fafa5528700 1 -- 172.21.15.27:0/987964446 shutdown_connections 2023-10-06T01:42:14.123 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.117+0000 7fafa5528700 1 -- 172.21.15.27:0/987964446 wait complete. 2023-10-06T01:42:14.125 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.118+0000 7fafa5528700 1 Processor -- start 2023-10-06T01:42:14.125 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.118+0000 7fafa5528700 1 -- start start 2023-10-06T01:42:14.125 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.119+0000 7fafa5528700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fafa00c6bf0 0x7fafa00691d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:14.125 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.119+0000 7fafa5528700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafa00cdc00 0x7fafa0145da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:14.125 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.119+0000 7fafa5528700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fafa00d1030 0x7fafa0067630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:14.126 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.119+0000 7fafa5528700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa0068d40 con 0x7fafa00cdc00 2023-10-06T01:42:14.126 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.119+0000 7fafa5528700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa0068130 con 0x7fafa00c6bf0 2023-10-06T01:42:14.126 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.119+0000 7fafa5528700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafa0068410 con 0x7fafa00d1030 2023-10-06T01:42:14.126 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.119+0000 7faf9e7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafa00cdc00 0x7fafa0145da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:14.126 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.119+0000 7faf9e7fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafa00cdc00 0x7fafa0145da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37690/0 (socket says 172.21.15.27:37690) 2023-10-06T01:42:14.126 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.119+0000 7faf9e7fc700 1 -- 172.21.15.27:0/3300688328 learned_addr learned my addr 172.21.15.27:0/3300688328 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:14.126 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.120+0000 7faf9f7fe700 1 --2- 172.21.15.27:0/3300688328 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fafa00d1030 0x7fafa0067630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:14.126 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.120+0000 7faf9effd700 1 --2- 172.21.15.27:0/3300688328 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fafa00c6bf0 0x7fafa00691d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:14.126 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.120+0000 7faf9e7fc700 1 -- 172.21.15.27:0/3300688328 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fafa00d1030 msgr2=0x7fafa0067630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:14.126 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.120+0000 7faf9e7fc700 1 --2- 172.21.15.27:0/3300688328 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fafa00d1030 0x7fafa0067630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:14.126 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.120+0000 7faf9e7fc700 1 -- 172.21.15.27:0/3300688328 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fafa00c6bf0 msgr2=0x7fafa00691d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:14.127 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.120+0000 7faf9e7fc700 1 --2- 172.21.15.27:0/3300688328 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fafa00c6bf0 0x7fafa00691d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:14.127 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.120+0000 7faf9e7fc700 1 -- 172.21.15.27:0/3300688328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf90009a20 con 0x7fafa00cdc00 2023-10-06T01:42:14.127 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.120+0000 7faf9e7fc700 1 --2- 172.21.15.27:0/3300688328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafa00cdc00 0x7fafa0145da0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7faf88009ed0 tx=0x7faf88016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:14.127 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.120+0000 7faf96ffd700 1 -- 172.21.15.27:0/3300688328 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf8801cc80 con 0x7fafa00cdc00 2023-10-06T01:42:14.127 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.120+0000 7faf96ffd700 1 -- 172.21.15.27:0/3300688328 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf8801ce20 con 0x7fafa00cdc00 2023-10-06T01:42:14.127 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.120+0000 7fafa5528700 1 -- 172.21.15.27:0/3300688328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf88009420 con 0x7fafa00cdc00 2023-10-06T01:42:14.127 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.120+0000 7faf96ffd700 1 -- 172.21.15.27:0/3300688328 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf8801a6a0 con 0x7fafa00cdc00 2023-10-06T01:42:14.127 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.120+0000 7fafa5528700 1 -- 172.21.15.27:0/3300688328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafa00689d0 con 0x7fafa00cdc00 2023-10-06T01:42:14.127 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.121+0000 7fafa5528700 1 -- 172.21.15.27:0/3300688328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafa0064050 con 0x7fafa00cdc00 2023-10-06T01:42:14.127 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.121+0000 7faf96ffd700 1 -- 172.21.15.27:0/3300688328 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7faf8801aa50 con 0x7fafa00cdc00 2023-10-06T01:42:14.127 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.122+0000 7faf96ffd700 1 --2- 172.21.15.27:0/3300688328 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7faf80066f30 0x7faf800693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:14.127 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.122+0000 7faf96ffd700 1 -- 172.21.15.27:0/3300688328 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6058+0+0 (secure 0 0 0) 0x7faf88085e40 con 0x7fafa00cdc00 2023-10-06T01:42:14.128 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.122+0000 7faf9effd700 1 --2- 172.21.15.27:0/3300688328 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7faf80066f30 0x7faf800693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:14.128 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.122+0000 7faf9effd700 1 --2- 172.21.15.27:0/3300688328 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7faf80066f30 0x7faf800693e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7faf900099f0 tx=0x7faf9000b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:14.128 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.126+0000 7faf96ffd700 1 -- 172.21.15.27:0/3300688328 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faf88053000 con 0x7fafa00cdc00 2023-10-06T01:42:14.367 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.366+0000 7fafa5528700 1 -- 172.21.15.27:0/3300688328 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafa00c5d50 con 0x7fafa00cdc00 2023-10-06T01:42:14.367 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.366+0000 7faf96ffd700 1 -- 172.21.15.27:0/3300688328 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4597 (secure 0 0 0) 0x7faf88029090 con 0x7fafa00cdc00 2023-10-06T01:42:14.368 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:14.368 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4870,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6836/1043044736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":1043044736},{"type":"v1","addr":"172.21.15.124:6836","nonce":1043044736}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4962,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/1207987117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1207987117},{"type":"v1","addr":"172.21.15.27:6834","nonce":1207987117}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4965,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/943261840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":943261840},{"type":"v1","addr":"172.21.15.27:6835","nonce":943261840}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T01:42:03.036784+0000","modified":"2023-10-06T01:42:06.652163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4974},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4974":{"gid":4974,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.124:6837/325247112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6835","nonce":325247112},{"type":"v1","addr":"172.21.15.124:6837","nonce":325247112}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-10-06T01:42:14.370 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.368+0000 7fafa5528700 1 -- 172.21.15.27:0/3300688328 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7faf80066f30 msgr2=0x7faf800693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:14.370 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.368+0000 7fafa5528700 1 --2- 172.21.15.27:0/3300688328 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7faf80066f30 0x7faf800693e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7faf900099f0 tx=0x7faf9000b040 comp rx=0 tx=0).stop 2023-10-06T01:42:14.370 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.368+0000 7fafa5528700 1 -- 172.21.15.27:0/3300688328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafa00cdc00 msgr2=0x7fafa0145da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:14.370 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.368+0000 7fafa5528700 1 --2- 172.21.15.27:0/3300688328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafa00cdc00 0x7fafa0145da0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7faf88009ed0 tx=0x7faf88016000 comp rx=0 tx=0).stop 2023-10-06T01:42:14.370 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.368+0000 7fafa5528700 1 -- 172.21.15.27:0/3300688328 shutdown_connections 2023-10-06T01:42:14.370 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.368+0000 7fafa5528700 1 --2- 172.21.15.27:0/3300688328 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7faf80066f30 0x7faf800693e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:14.370 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.368+0000 7fafa5528700 1 --2- 172.21.15.27:0/3300688328 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fafa00d1030 0x7fafa0067630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:14.371 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.368+0000 7fafa5528700 1 --2- 172.21.15.27:0/3300688328 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fafa00c6bf0 0x7fafa00691d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:14.371 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.368+0000 7fafa5528700 1 --2- 172.21.15.27:0/3300688328 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fafa00cdc00 0x7fafa0145da0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:14.371 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.368+0000 7fafa5528700 1 -- 172.21.15.27:0/3300688328 >> 172.21.15.27:0/3300688328 conn(0x7fafa0099990 msgr2=0x7fafa00c97b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:14.371 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.369+0000 7fafa5528700 1 -- 172.21.15.27:0/3300688328 shutdown_connections 2023-10-06T01:42:14.371 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.369+0000 7fafa5528700 1 -- 172.21.15.27:0/3300688328 wait complete. 2023-10-06T01:42:14.371 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 15 2023-10-06T01:42:14.382 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json flush journal 2023-10-06T01:42:14.536 INFO:teuthology.orchestra.run.smithi124.stdout:{"message":"","return_code":0} 2023-10-06T01:42:14.536 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2023-10-06T01:42:14.537 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001fb.00000000 parent 2023-10-06T01:42:14.576 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.574+0000 7f99573da000 1 Processor -- start 2023-10-06T01:42:14.576 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.574+0000 7f99573da000 1 -- start start 2023-10-06T01:42:14.578 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.575+0000 7f99573da000 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55d64cfc8c40 0x55d64d081ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:14.578 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.575+0000 7f99573da000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d64d082ad0 0x55d64d099060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:14.578 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.575+0000 7f99573da000 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55d64d082ea0 0x55d64cfc74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:14.578 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.575+0000 7f99573da000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d64cf10980 con 0x55d64d082ad0 2023-10-06T01:42:14.578 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.575+0000 7f99573da000 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d64cf10620 con 0x55d64d082ea0 2023-10-06T01:42:14.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.575+0000 7f99573da000 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d64cf07400 con 0x55d64cfc8c40 2023-10-06T01:42:14.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.575+0000 7f994fe14700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d64d082ad0 0x55d64d099060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:14.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.575+0000 7f994fe14700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d64d082ad0 0x55d64d099060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37708/0 (socket says 172.21.15.27:37708) 2023-10-06T01:42:14.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.575+0000 7f994fe14700 1 -- 172.21.15.27:0/203486720 learned_addr learned my addr 172.21.15.27:0/203486720 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:14.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.575+0000 7f9950e16700 1 --2- 172.21.15.27:0/203486720 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55d64d082ea0 0x55d64cfc74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:14.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.575+0000 7f994fe14700 1 -- 172.21.15.27:0/203486720 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55d64cfc8c40 msgr2=0x55d64d081ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:14.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.575+0000 7f9950615700 1 --2- 172.21.15.27:0/203486720 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55d64cfc8c40 0x55d64d081ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:14.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.575+0000 7f994fe14700 1 --2- 172.21.15.27:0/203486720 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55d64cfc8c40 0x55d64d081ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:14.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.575+0000 7f994fe14700 1 -- 172.21.15.27:0/203486720 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55d64d082ea0 msgr2=0x55d64cfc74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:14.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.575+0000 7f994fe14700 1 --2- 172.21.15.27:0/203486720 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55d64d082ea0 0x55d64cfc74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:14.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.575+0000 7f994fe14700 1 -- 172.21.15.27:0/203486720 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994000c3d0 con 0x55d64d082ad0 2023-10-06T01:42:14.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.575+0000 7f994fe14700 1 --2- 172.21.15.27:0/203486720 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d64d082ad0 0x55d64d099060 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f993800b260 tx=0x7f99380017b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee6f45efdcf9166 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:14.580 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.576+0000 7f994ee12700 1 -- 172.21.15.27:0/203486720 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9938014070 con 0x55d64d082ad0 2023-10-06T01:42:14.580 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.576+0000 7f994ee12700 1 -- 172.21.15.27:0/203486720 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993800be80 con 0x55d64d082ad0 2023-10-06T01:42:14.580 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.576+0000 7f994ee12700 1 -- 172.21.15.27:0/203486720 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993800f680 con 0x55d64d082ad0 2023-10-06T01:42:14.580 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.576+0000 7f99573da000 1 -- 172.21.15.27:0/203486720 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d64d082ad0 msgr2=0x55d64d099060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:14.580 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.576+0000 7f99573da000 1 --2- 172.21.15.27:0/203486720 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d64d082ad0 0x55d64d099060 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f993800b260 tx=0x7f99380017b0 comp rx=0 tx=0).stop 2023-10-06T01:42:14.580 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.576+0000 7f99573da000 1 -- 172.21.15.27:0/203486720 shutdown_connections 2023-10-06T01:42:14.580 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.576+0000 7f99573da000 1 --2- 172.21.15.27:0/203486720 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55d64cfc8c40 0x55d64d081ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:14.580 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.576+0000 7f99573da000 1 --2- 172.21.15.27:0/203486720 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55d64d082ea0 0x55d64cfc74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:14.580 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.576+0000 7f99573da000 1 --2- 172.21.15.27:0/203486720 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d64d082ad0 0x55d64d099060 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:14.580 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.576+0000 7f99573da000 1 -- 172.21.15.27:0/203486720 >> 172.21.15.27:0/203486720 conn(0x55d64cfc7120 msgr2=0x55d64cfc8240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:14.580 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.576+0000 7f99573da000 1 -- 172.21.15.27:0/203486720 shutdown_connections 2023-10-06T01:42:14.580 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.576+0000 7f99573da000 1 -- 172.21.15.27:0/203486720 wait complete. 2023-10-06T01:42:14.580 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.577+0000 7f99573da000 1 Processor -- start 2023-10-06T01:42:14.581 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.577+0000 7f99573da000 1 -- start start 2023-10-06T01:42:14.584 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.577+0000 7f99573da000 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55d64cfc8c40 0x55d64d00c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:14.584 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.577+0000 7f99573da000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d64d082ad0 0x55d64d0073c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:14.584 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.577+0000 7f99573da000 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55d64d082ea0 0x55d64d096c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:14.584 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.577+0000 7f99573da000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d64d007900 con 0x55d64d082ad0 2023-10-06T01:42:14.584 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.577+0000 7f99573da000 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d64d007b40 con 0x55d64d082ea0 2023-10-06T01:42:14.585 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.577+0000 7f99573da000 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d64d007de0 con 0x55d64cfc8c40 2023-10-06T01:42:14.585 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.578+0000 7f994fe14700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d64d082ad0 0x55d64d0073c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:14.585 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.578+0000 7f994fe14700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d64d082ad0 0x55d64d0073c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37718/0 (socket says 172.21.15.27:37718) 2023-10-06T01:42:14.585 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.578+0000 7f994fe14700 1 -- 172.21.15.27:0/3152205737 learned_addr learned my addr 172.21.15.27:0/3152205737 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:14.585 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.578+0000 7f9950615700 1 --2- 172.21.15.27:0/3152205737 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55d64cfc8c40 0x55d64d00c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:14.585 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.578+0000 7f9950e16700 1 --2- 172.21.15.27:0/3152205737 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55d64d082ea0 0x55d64d096c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:14.585 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.578+0000 7f994fe14700 1 -- 172.21.15.27:0/3152205737 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55d64cfc8c40 msgr2=0x55d64d00c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:14.585 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.578+0000 7f994fe14700 1 --2- 172.21.15.27:0/3152205737 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55d64cfc8c40 0x55d64d00c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:14.585 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.578+0000 7f994fe14700 1 -- 172.21.15.27:0/3152205737 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55d64d082ea0 msgr2=0x55d64d096c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:14.585 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.578+0000 7f994fe14700 1 --2- 172.21.15.27:0/3152205737 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55d64d082ea0 0x55d64d096c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:14.585 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.578+0000 7f994fe14700 1 -- 172.21.15.27:0/3152205737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9938001a90 con 0x55d64d082ad0 2023-10-06T01:42:14.585 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.578+0000 7f994fe14700 1 --2- 172.21.15.27:0/3152205737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d64d082ad0 0x55d64d0073c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f993800b9b0 tx=0x7f9938016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:14.585 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.578+0000 7f994d60f700 1 -- 172.21.15.27:0/3152205737 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993801c7a0 con 0x55d64d082ad0 2023-10-06T01:42:14.586 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.578+0000 7f994d60f700 1 -- 172.21.15.27:0/3152205737 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993801c900 con 0x55d64d082ad0 2023-10-06T01:42:14.586 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.578+0000 7f99573da000 1 -- 172.21.15.27:0/3152205737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55d64cf1f3d0 con 0x55d64d082ad0 2023-10-06T01:42:14.586 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.578+0000 7f994d60f700 1 -- 172.21.15.27:0/3152205737 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f993801ca70 con 0x55d64d082ad0 2023-10-06T01:42:14.586 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.578+0000 7f99573da000 1 -- 172.21.15.27:0/3152205737 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d64cf12ae0 con 0x55d64d082ad0 2023-10-06T01:42:14.586 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.579+0000 7f994d60f700 1 -- 172.21.15.27:0/3152205737 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9938023020 con 0x55d64d082ad0 2023-10-06T01:42:14.586 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.580+0000 7f994d60f700 1 --2- 172.21.15.27:0/3152205737 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9930065890 0x7f9930067d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:14.586 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.580+0000 7f994d60f700 1 -- 172.21.15.27:0/3152205737 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f9938008030 con 0x55d64d082ad0 2023-10-06T01:42:14.586 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.580+0000 7f9950615700 1 --2- 172.21.15.27:0/3152205737 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9930065890 0x7f9930067d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:14.586 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.580+0000 7f99573da000 1 --2- 172.21.15.27:0/3152205737 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x55d64d097140 0x55d64d008a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:14.586 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.580+0000 7f99573da000 1 -- 172.21.15.27:0/3152205737 --> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] -- osd_op(unknown.0.0:1 6.13 6:ca8fdbba:::100000001fb.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) v8 -- 0x55d64d008fa0 con 0x55d64d097140 2023-10-06T01:42:14.586 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.580+0000 7f9950e16700 1 --2- 172.21.15.27:0/3152205737 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x55d64d097140 0x55d64d008a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.580+0000 7f9950615700 1 --2- 172.21.15.27:0/3152205737 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9930065890 0x7f9930067d40 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9948009b40 tx=0x7f9948005a50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.581+0000 7f9950e16700 1 --2- 172.21.15.27:0/3152205737 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x55d64d097140 0x55d64d008a60 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.581+0000 7f9950e16700 1 -- 172.21.15.27:0/3152205737 <== osd.3 v2:172.21.15.27:6824/90668 1 ==== osd_op_reply(1 100000001fb.00000000 [getxattr (95) out=95b] v0'0 uv1 ondisk = 0) v8 ==== 164+0+95 (crc 0 0 0) 0x7f994000ab60 con 0x55d64d097140 2023-10-06T01:42:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.581+0000 7f99573da000 1 -- 172.21.15.27:0/3152205737 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x55d64d097140 msgr2=0x55d64d008a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.581+0000 7f99573da000 1 --2- 172.21.15.27:0/3152205737 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x55d64d097140 0x55d64d008a60 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.581+0000 7f99573da000 1 -- 172.21.15.27:0/3152205737 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9930065890 msgr2=0x7f9930067d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.581+0000 7f99573da000 1 --2- 172.21.15.27:0/3152205737 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9930065890 0x7f9930067d40 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9948009b40 tx=0x7f9948005a50 comp rx=0 tx=0).stop 2023-10-06T01:42:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.581+0000 7f99573da000 1 -- 172.21.15.27:0/3152205737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d64d082ad0 msgr2=0x55d64d0073c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.581+0000 7f99573da000 1 --2- 172.21.15.27:0/3152205737 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55d64d082ad0 0x55d64d0073c0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f993800b9b0 tx=0x7f9938016000 comp rx=0 tx=0).stop 2023-10-06T01:42:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.581+0000 7f9950e16700 1 -- 172.21.15.27:0/3152205737 reap_dead start 2023-10-06T01:42:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.581+0000 7f99573da000 1 -- 172.21.15.27:0/3152205737 shutdown_connections 2023-10-06T01:42:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.581+0000 7f99573da000 1 -- 172.21.15.27:0/3152205737 >> 172.21.15.27:0/3152205737 conn(0x55d64cfc7120 msgr2=0x55d64d00d980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:14.587 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.581+0000 7f99573da000 1 -- 172.21.15.27:0/3152205737 shutdown_connections 2023-10-06T01:42:14.588 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.581+0000 7f99573da000 1 -- 172.21.15.27:0/3152205737 wait complete. 2023-10-06T01:42:14.588 INFO:teuthology.orchestra.run.smithi027.stdout:Y��alphaparent_a 2023-10-06T01:42:14.589 DEBUG:teuthology.orchestra.run.smithi027:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2023-10-06T01:42:14.681 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2023-10-06T01:42:14.681 INFO:teuthology.orchestra.run.smithi027.stdout: "ino": 1099511628283, 2023-10-06T01:42:14.681 INFO:teuthology.orchestra.run.smithi027.stdout: "ancestors": [ 2023-10-06T01:42:14.681 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:42:14.681 INFO:teuthology.orchestra.run.smithi027.stdout: "dirino": 1099511628282, 2023-10-06T01:42:14.681 INFO:teuthology.orchestra.run.smithi027.stdout: "dname": "alpha", 2023-10-06T01:42:14.681 INFO:teuthology.orchestra.run.smithi027.stdout: "version": 2 2023-10-06T01:42:14.681 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2023-10-06T01:42:14.682 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:42:14.682 INFO:teuthology.orchestra.run.smithi027.stdout: "dirino": 1, 2023-10-06T01:42:14.682 INFO:teuthology.orchestra.run.smithi027.stdout: "dname": "parent_a", 2023-10-06T01:42:14.682 INFO:teuthology.orchestra.run.smithi027.stdout: "version": 18 2023-10-06T01:42:14.682 INFO:teuthology.orchestra.run.smithi027.stdout: } 2023-10-06T01:42:14.682 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2023-10-06T01:42:14.682 INFO:teuthology.orchestra.run.smithi027.stdout: "pool": 6, 2023-10-06T01:42:14.682 INFO:teuthology.orchestra.run.smithi027.stdout: "old_pools": [] 2023-10-06T01:42:14.682 INFO:teuthology.orchestra.run.smithi027.stdout:} 2023-10-06T01:42:14.682 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:14.685 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001fb.00000000 layout 2023-10-06T01:42:14.764 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.763+0000 7fb3392fb000 1 Processor -- start 2023-10-06T01:42:14.764 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.763+0000 7fb3392fb000 1 -- start start 2023-10-06T01:42:14.771 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.763+0000 7fb3392fb000 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55a4f7b29ad0 0x55a4f7bfe1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:14.771 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.763+0000 7fb3392fb000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f7bfbf90 0x55a4f7b58200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:14.771 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.763+0000 7fb3392fb000 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55a4f7bfd370 0x55a4f7bf5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:14.771 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.763+0000 7fb3392fb000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4f7ace320 con 0x55a4f7bfbf90 2023-10-06T01:42:14.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.763+0000 7fb3392fb000 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4f7a72980 con 0x55a4f7bfd370 2023-10-06T01:42:14.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.763+0000 7fb3392fb000 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a4f7a72620 con 0x55a4f7b29ad0 2023-10-06T01:42:14.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.763+0000 7fb331d35700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f7bfbf90 0x55a4f7b58200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:14.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.763+0000 7fb331d35700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f7bfbf90 0x55a4f7b58200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37734/0 (socket says 172.21.15.27:37734) 2023-10-06T01:42:14.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.763+0000 7fb331d35700 1 -- 172.21.15.27:0/4067848564 learned_addr learned my addr 172.21.15.27:0/4067848564 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:14.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.763+0000 7fb332d37700 1 --2- 172.21.15.27:0/4067848564 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55a4f7bfd370 0x55a4f7bf5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:14.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.763+0000 7fb332536700 1 --2- 172.21.15.27:0/4067848564 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55a4f7b29ad0 0x55a4f7bfe1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:14.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.764+0000 7fb331d35700 1 -- 172.21.15.27:0/4067848564 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55a4f7b29ad0 msgr2=0x55a4f7bfe1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:14.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.764+0000 7fb331d35700 1 --2- 172.21.15.27:0/4067848564 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55a4f7b29ad0 0x55a4f7bfe1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:14.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.764+0000 7fb331d35700 1 -- 172.21.15.27:0/4067848564 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55a4f7bfd370 msgr2=0x55a4f7bf5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:14.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.764+0000 7fb331d35700 1 --2- 172.21.15.27:0/4067848564 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55a4f7bfd370 0x55a4f7bf5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:14.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.764+0000 7fb331d35700 1 -- 172.21.15.27:0/4067848564 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb31c005cf0 con 0x55a4f7bfbf90 2023-10-06T01:42:14.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.764+0000 7fb331d35700 1 --2- 172.21.15.27:0/4067848564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f7bfbf90 0x55a4f7b58200 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x55a4f7b26560 tx=0x7fb31c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=545776f523291eee server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:14.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.764+0000 7fb330d33700 1 -- 172.21.15.27:0/4067848564 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31c002f70 con 0x55a4f7bfbf90 2023-10-06T01:42:14.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.764+0000 7fb330d33700 1 -- 172.21.15.27:0/4067848564 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb31c000bc0 con 0x55a4f7bfbf90 2023-10-06T01:42:14.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.764+0000 7fb330d33700 1 -- 172.21.15.27:0/4067848564 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb31c007b10 con 0x55a4f7bfbf90 2023-10-06T01:42:14.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.764+0000 7fb3392fb000 1 -- 172.21.15.27:0/4067848564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f7bfbf90 msgr2=0x55a4f7b58200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:14.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.764+0000 7fb3392fb000 1 --2- 172.21.15.27:0/4067848564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f7bfbf90 0x55a4f7b58200 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x55a4f7b26560 tx=0x7fb31c015000 comp rx=0 tx=0).stop 2023-10-06T01:42:14.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.764+0000 7fb3392fb000 1 -- 172.21.15.27:0/4067848564 shutdown_connections 2023-10-06T01:42:14.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.764+0000 7fb3392fb000 1 --2- 172.21.15.27:0/4067848564 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55a4f7b29ad0 0x55a4f7bfe1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:14.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.764+0000 7fb3392fb000 1 --2- 172.21.15.27:0/4067848564 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55a4f7bfd370 0x55a4f7bf5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:14.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.764+0000 7fb3392fb000 1 --2- 172.21.15.27:0/4067848564 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f7bfbf90 0x55a4f7b58200 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:14.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.764+0000 7fb3392fb000 1 -- 172.21.15.27:0/4067848564 >> 172.21.15.27:0/4067848564 conn(0x55a4f7b57bc0 msgr2=0x55a4f7bfa700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:14.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.764+0000 7fb3392fb000 1 -- 172.21.15.27:0/4067848564 shutdown_connections 2023-10-06T01:42:14.774 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.765+0000 7fb3392fb000 1 -- 172.21.15.27:0/4067848564 wait complete. 2023-10-06T01:42:14.774 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.765+0000 7fb3392fb000 1 Processor -- start 2023-10-06T01:42:14.774 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.765+0000 7fb3392fb000 1 -- start start 2023-10-06T01:42:14.774 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.766+0000 7fb3392fb000 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55a4f7b29ad0 0x55a4f7bfae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:14.774 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.766+0000 7fb3392fb000 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55a4f7bfbf90 0x55a4f7b4b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:14.774 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.766+0000 7fb3392fb000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f7bfd370 0x55a4f7b54770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:14.774 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.766+0000 7fb3392fb000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4f7a69400 con 0x55a4f7bfd370 2023-10-06T01:42:14.774 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.766+0000 7fb3392fb000 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a4f7b273d0 con 0x55a4f7b29ad0 2023-10-06T01:42:14.774 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.766+0000 7fb3392fb000 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a4f7b278a0 con 0x55a4f7bfbf90 2023-10-06T01:42:14.774 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.766+0000 7fb332d37700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f7bfd370 0x55a4f7b54770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:14.774 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.766+0000 7fb332d37700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f7bfd370 0x55a4f7b54770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37736/0 (socket says 172.21.15.27:37736) 2023-10-06T01:42:14.774 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.766+0000 7fb332d37700 1 -- 172.21.15.27:0/1234818733 learned_addr learned my addr 172.21.15.27:0/1234818733 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:14.775 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.766+0000 7fb332536700 1 --2- 172.21.15.27:0/1234818733 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55a4f7b29ad0 0x55a4f7bfae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:14.775 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.766+0000 7fb331d35700 1 --2- 172.21.15.27:0/1234818733 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55a4f7bfbf90 0x55a4f7b4b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:14.775 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.766+0000 7fb332d37700 1 -- 172.21.15.27:0/1234818733 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55a4f7bfbf90 msgr2=0x55a4f7b4b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:14.775 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.766+0000 7fb332d37700 1 --2- 172.21.15.27:0/1234818733 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55a4f7bfbf90 0x55a4f7b4b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:14.775 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.766+0000 7fb332d37700 1 -- 172.21.15.27:0/1234818733 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55a4f7b29ad0 msgr2=0x55a4f7bfae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:14.775 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.766+0000 7fb332d37700 1 --2- 172.21.15.27:0/1234818733 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55a4f7b29ad0 0x55a4f7bfae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:14.775 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.766+0000 7fb332d37700 1 -- 172.21.15.27:0/1234818733 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a4f7a813d0 con 0x55a4f7bfd370 2023-10-06T01:42:14.775 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.766+0000 7fb332d37700 1 --2- 172.21.15.27:0/1234818733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f7bfd370 0x55a4f7b54770 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fb32400d490 tx=0x7fb324006910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:14.775 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.767+0000 7fb32affd700 1 -- 172.21.15.27:0/1234818733 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb324006bd0 con 0x55a4f7bfd370 2023-10-06T01:42:14.775 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.767+0000 7fb32affd700 1 -- 172.21.15.27:0/1234818733 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb32400dea0 con 0x55a4f7bfd370 2023-10-06T01:42:14.775 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.767+0000 7fb3392fb000 1 -- 172.21.15.27:0/1234818733 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55a4f7a74ae0 con 0x55a4f7bfd370 2023-10-06T01:42:14.775 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.767+0000 7fb3392fb000 1 -- 172.21.15.27:0/1234818733 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a4f7a74540 con 0x55a4f7bfd370 2023-10-06T01:42:14.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.767+0000 7fb32affd700 1 -- 172.21.15.27:0/1234818733 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3240136d0 con 0x55a4f7bfd370 2023-10-06T01:42:14.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.768+0000 7fb32affd700 1 -- 172.21.15.27:0/1234818733 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb324013b00 con 0x55a4f7bfd370 2023-10-06T01:42:14.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.768+0000 7fb32affd700 1 --2- 172.21.15.27:0/1234818733 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb314065890 0x7fb314067d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:14.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.769+0000 7fb32affd700 1 -- 172.21.15.27:0/1234818733 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fb324083440 con 0x55a4f7bfd370 2023-10-06T01:42:14.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.769+0000 7fb332536700 1 --2- 172.21.15.27:0/1234818733 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb314065890 0x7fb314067d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:14.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.769+0000 7fb3392fb000 1 --2- 172.21.15.27:0/1234818733 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x55a4f7c0f2d0 0x55a4f7b284f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:14.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.769+0000 7fb3392fb000 1 -- 172.21.15.27:0/1234818733 --> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] -- osd_op(unknown.0.0:1 6.13 6:ca8fdbba:::100000001fb.00000000:head [getxattr layout in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) v8 -- 0x55a4f7b6e920 con 0x55a4f7c0f2d0 2023-10-06T01:42:14.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.769+0000 7fb331d35700 1 --2- 172.21.15.27:0/1234818733 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x55a4f7c0f2d0 0x55a4f7b284f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:14.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.769+0000 7fb332536700 1 --2- 172.21.15.27:0/1234818733 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb314065890 0x7fb314067d40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb32c004170 tx=0x7fb32c000bc0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:14.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.769+0000 7fb331d35700 1 --2- 172.21.15.27:0/1234818733 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x55a4f7c0f2d0 0x55a4f7b284f0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:14.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.769+0000 7fb331d35700 1 -- 172.21.15.27:0/1234818733 <== osd.3 v2:172.21.15.27:6824/90668 1 ==== osd_op_reply(1 100000001fb.00000000 [getxattr (30) out=30b] v0'0 uv1 ondisk = 0) v8 ==== 164+0+30 (crc 0 0 0) 0x7fb31c0078e0 con 0x55a4f7c0f2d0 2023-10-06T01:42:14.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.769+0000 7fb3392fb000 1 -- 172.21.15.27:0/1234818733 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x55a4f7c0f2d0 msgr2=0x55a4f7b284f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:14.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.769+0000 7fb3392fb000 1 --2- 172.21.15.27:0/1234818733 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x55a4f7c0f2d0 0x55a4f7b284f0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:14.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.769+0000 7fb3392fb000 1 -- 172.21.15.27:0/1234818733 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb314065890 msgr2=0x7fb314067d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:14.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.769+0000 7fb3392fb000 1 --2- 172.21.15.27:0/1234818733 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb314065890 0x7fb314067d40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb32c004170 tx=0x7fb32c000bc0 comp rx=0 tx=0).stop 2023-10-06T01:42:14.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.770+0000 7fb3392fb000 1 -- 172.21.15.27:0/1234818733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f7bfd370 msgr2=0x55a4f7b54770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:14.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.770+0000 7fb3392fb000 1 --2- 172.21.15.27:0/1234818733 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a4f7bfd370 0x55a4f7b54770 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fb32400d490 tx=0x7fb324006910 comp rx=0 tx=0).stop 2023-10-06T01:42:14.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.770+0000 7fb332d37700 1 -- 172.21.15.27:0/1234818733 reap_dead start 2023-10-06T01:42:14.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.770+0000 7fb3392fb000 1 -- 172.21.15.27:0/1234818733 shutdown_connections 2023-10-06T01:42:14.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.770+0000 7fb3392fb000 1 -- 172.21.15.27:0/1234818733 >> 172.21.15.27:0/1234818733 conn(0x55a4f7b57bc0 msgr2=0x55a4f7b58200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:14.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.770+0000 7fb3392fb000 1 -- 172.21.15.27:0/1234818733 shutdown_connections 2023-10-06T01:42:14.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:14.770+0000 7fb3392fb000 1 -- 172.21.15.27:0/1234818733 wait complete. 2023-10-06T01:42:14.778 INFO:teuthology.orchestra.run.smithi027.stdout:@@ 2023-10-06T01:42:14.778 DEBUG:teuthology.orchestra.run.smithi027:> ceph-dencoder type file_layout_t import - decode dump_json 2023-10-06T01:42:14.871 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2023-10-06T01:42:14.872 INFO:teuthology.orchestra.run.smithi027.stdout: "stripe_unit": 4194304, 2023-10-06T01:42:14.872 INFO:teuthology.orchestra.run.smithi027.stdout: "stripe_count": 1, 2023-10-06T01:42:14.872 INFO:teuthology.orchestra.run.smithi027.stdout: "object_size": 4194304, 2023-10-06T01:42:14.873 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_id": 6, 2023-10-06T01:42:14.873 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_ns": "" 2023-10-06T01:42:14.873 INFO:teuthology.orchestra.run.smithi027.stdout:} 2023-10-06T01:42:14.873 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:14.875 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T01:42:14.875 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parent_b) 2023-10-06T01:42:14.932 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T01:42:14.932 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mv parent_a/alpha parent_b/alpha) 2023-10-06T01:42:14.991 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T01:42:15.150 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.148+0000 7fc561535700 1 Processor -- start 2023-10-06T01:42:15.150 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.148+0000 7fc561535700 1 -- start start 2023-10-06T01:42:15.159 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.149+0000 7fc561535700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc55c159950 0x7fc55c0b0190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:15.159 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.149+0000 7fc561535700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc55c0ad3c0 con 0x7fc55c0ae860 2023-10-06T01:42:15.159 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.149+0000 7fc561535700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc55c0b0ec0 con 0x7fc55c159580 2023-10-06T01:42:15.159 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.149+0000 7fc561535700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc55c0b11a0 con 0x7fc55c159950 2023-10-06T01:42:15.159 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.149+0000 7fc55b7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fc55c0ae860 0x7fc55c0b06d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:52196/0 (socket says 172.21.15.27:52196) 2023-10-06T01:42:15.159 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.149+0000 7fc55b7fe700 1 -- 172.21.15.27:0/1085925295 learned_addr learned my addr 172.21.15.27:0/1085925295 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:15.160 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.149+0000 7fc55a7fc700 1 --2- 172.21.15.27:0/1085925295 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc55c159950 0x7fc55c0b0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:15.160 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.149+0000 7fc559ffb700 1 -- 172.21.15.27:0/1085925295 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366092793 0 0) 0x7fc55c0ad3c0 con 0x7fc55c0ae860 2023-10-06T01:42:15.160 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.149+0000 7fc559ffb700 1 -- 172.21.15.27:0/1085925295 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc548003570 con 0x7fc55c0ae860 2023-10-06T01:42:15.160 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.150+0000 7fc559ffb700 1 -- 172.21.15.27:0/1085925295 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3962251426 0 0) 0x7fc55c0b0ec0 con 0x7fc55c159580 2023-10-06T01:42:15.160 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.150+0000 7fc559ffb700 1 -- 172.21.15.27:0/1085925295 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc55c0ad3c0 con 0x7fc55c159580 2023-10-06T01:42:15.160 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.150+0000 7fc55a7fc700 1 -- 172.21.15.27:0/1085925295 >> v1:172.21.15.124:6789/0 conn(0x7fc55c159580 legacy=0x7fc55c0b2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:15.161 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.150+0000 7fc559ffb700 1 -- 172.21.15.27:0/1085925295 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1948232530 0 0) 0x7fc548003570 con 0x7fc55c0ae860 2023-10-06T01:42:15.161 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.150+0000 7fc55a7fc700 1 -- 172.21.15.27:0/1085925295 >> v1:172.21.15.27:6789/0 conn(0x7fc55c0ae860 legacy=0x7fc55c0b06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:15.161 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.150+0000 7fc55a7fc700 1 -- 172.21.15.27:0/1085925295 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc55c0acd70 con 0x7fc55c159950 2023-10-06T01:42:15.161 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.150+0000 7fc55a7fc700 1 --2- 172.21.15.27:0/1085925295 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc55c159950 0x7fc55c0b0190 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc544009910 tx=0x7fc5440091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1641167e3d53948c server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:15.161 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.150+0000 7fc559ffb700 1 -- 172.21.15.27:0/1085925295 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc54c001470 con 0x7fc55c159950 2023-10-06T01:42:15.161 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.150+0000 7fc559ffb700 1 -- 172.21.15.27:0/1085925295 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc544011420 con 0x7fc55c159950 2023-10-06T01:42:15.162 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.150+0000 7fc559ffb700 1 -- 172.21.15.27:0/1085925295 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5440117c0 con 0x7fc55c159950 2023-10-06T01:42:15.162 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.150+0000 7fc561535700 1 -- 172.21.15.27:0/1085925295 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc55c159950 msgr2=0x7fc55c0b0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:15.162 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.150+0000 7fc561535700 1 --2- 172.21.15.27:0/1085925295 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc55c159950 0x7fc55c0b0190 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc544009910 tx=0x7fc5440091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:15.162 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.151+0000 7fc561535700 1 -- 172.21.15.27:0/1085925295 shutdown_connections 2023-10-06T01:42:15.162 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.151+0000 7fc561535700 1 --2- 172.21.15.27:0/1085925295 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc55c159950 0x7fc55c0b0190 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:15.162 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.151+0000 7fc561535700 1 -- 172.21.15.27:0/1085925295 >> 172.21.15.27:0/1085925295 conn(0x7fc55c06e6f0 msgr2=0x7fc55c06afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:15.163 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.151+0000 7fc561535700 1 -- 172.21.15.27:0/1085925295 shutdown_connections 2023-10-06T01:42:15.163 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.151+0000 7fc561535700 1 -- 172.21.15.27:0/1085925295 wait complete. 2023-10-06T01:42:15.163 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.152+0000 7fc561535700 1 Processor -- start 2023-10-06T01:42:15.163 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.152+0000 7fc561535700 1 -- start start 2023-10-06T01:42:15.163 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.153+0000 7fc561535700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc55c0ae860 0x7fc55c0aa290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:15.163 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.153+0000 7fc561535700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc55c159580 0x7fc55c153540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:15.164 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.153+0000 7fc561535700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc55c159950 0x7fc55c15bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:15.164 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.153+0000 7fc561535700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc55c1530b0 con 0x7fc55c0ae860 2023-10-06T01:42:15.164 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.153+0000 7fc561535700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc55c15c2b0 con 0x7fc55c159580 2023-10-06T01:42:15.164 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.153+0000 7fc561535700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc55c1524a0 con 0x7fc55c159950 2023-10-06T01:42:15.164 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.153+0000 7fc55affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc55c0ae860 0x7fc55c0aa290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:15.164 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.153+0000 7fc55affd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc55c0ae860 0x7fc55c0aa290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:37738/0 (socket says 172.21.15.27:37738) 2023-10-06T01:42:15.165 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.153+0000 7fc55affd700 1 -- 172.21.15.27:0/2402978917 learned_addr learned my addr 172.21.15.27:0/2402978917 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:15.165 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.153+0000 7fc55b7fe700 1 --2- 172.21.15.27:0/2402978917 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc55c159950 0x7fc55c15bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:15.165 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.153+0000 7fc55a7fc700 1 --2- 172.21.15.27:0/2402978917 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc55c159580 0x7fc55c153540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:15.165 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.153+0000 7fc55affd700 1 -- 172.21.15.27:0/2402978917 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc55c159950 msgr2=0x7fc55c15bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:15.165 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.153+0000 7fc55affd700 1 --2- 172.21.15.27:0/2402978917 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc55c159950 0x7fc55c15bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:15.166 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.153+0000 7fc55affd700 1 -- 172.21.15.27:0/2402978917 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc55c159580 msgr2=0x7fc55c153540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:15.166 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.153+0000 7fc55affd700 1 --2- 172.21.15.27:0/2402978917 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc55c159580 0x7fc55c153540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:15.166 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.153+0000 7fc55affd700 1 -- 172.21.15.27:0/2402978917 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc544009420 con 0x7fc55c0ae860 2023-10-06T01:42:15.166 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.153+0000 7fc55b7fe700 1 --2- 172.21.15.27:0/2402978917 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc55c159950 0x7fc55c15bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:42:15.166 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.153+0000 7fc55affd700 1 --2- 172.21.15.27:0/2402978917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc55c0ae860 0x7fc55c0aa290 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fc55400cb10 tx=0x7fc554002820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:15.166 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.153+0000 7fc552ffd700 1 -- 172.21.15.27:0/2402978917 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55400a760 con 0x7fc55c0ae860 2023-10-06T01:42:15.167 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.153+0000 7fc552ffd700 1 -- 172.21.15.27:0/2402978917 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc55400a8c0 con 0x7fc55c0ae860 2023-10-06T01:42:15.167 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.153+0000 7fc561535700 1 -- 172.21.15.27:0/2402978917 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc55c152780 con 0x7fc55c0ae860 2023-10-06T01:42:15.167 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.153+0000 7fc552ffd700 1 -- 172.21.15.27:0/2402978917 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc55400ac60 con 0x7fc55c0ae860 2023-10-06T01:42:15.167 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.153+0000 7fc561535700 1 -- 172.21.15.27:0/2402978917 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc55c152d40 con 0x7fc55c0ae860 2023-10-06T01:42:15.167 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.154+0000 7fc561535700 1 -- 172.21.15.27:0/2402978917 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc55c064050 con 0x7fc55c0ae860 2023-10-06T01:42:15.167 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.155+0000 7fc552ffd700 1 -- 172.21.15.27:0/2402978917 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc554012630 con 0x7fc55c0ae860 2023-10-06T01:42:15.168 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.155+0000 7fc552ffd700 1 --2- 172.21.15.27:0/2402978917 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc53c066f30 0x7fc53c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:15.168 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.155+0000 7fc552ffd700 1 -- 172.21.15.27:0/2402978917 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fc554084a50 con 0x7fc55c0ae860 2023-10-06T01:42:15.168 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.155+0000 7fc55a7fc700 1 --2- 172.21.15.27:0/2402978917 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc53c066f30 0x7fc53c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:15.168 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.156+0000 7fc55a7fc700 1 --2- 172.21.15.27:0/2402978917 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc53c066f30 0x7fc53c0693e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc55c159da0 tx=0x7fc544016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:15.169 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.160+0000 7fc552ffd700 1 -- 172.21.15.27:0/2402978917 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc554051b60 con 0x7fc55c0ae860 2023-10-06T01:42:15.397 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.395+0000 7fc561535700 1 -- 172.21.15.27:0/2402978917 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc55c153390 con 0x7fc55c0ae860 2023-10-06T01:42:15.398 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.396+0000 7fc552ffd700 1 -- 172.21.15.27:0/2402978917 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4597 (secure 0 0 0) 0x7fc554055180 con 0x7fc55c0ae860 2023-10-06T01:42:15.398 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:15.398 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4870,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6836/1043044736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":1043044736},{"type":"v1","addr":"172.21.15.124:6836","nonce":1043044736}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4962,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/1207987117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1207987117},{"type":"v1","addr":"172.21.15.27:6834","nonce":1207987117}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4965,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/943261840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":943261840},{"type":"v1","addr":"172.21.15.27:6835","nonce":943261840}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T01:42:03.036784+0000","modified":"2023-10-06T01:42:06.652163+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4974},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4974":{"gid":4974,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.124:6837/325247112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6835","nonce":325247112},{"type":"v1","addr":"172.21.15.124:6837","nonce":325247112}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-10-06T01:42:15.400 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.398+0000 7fc561535700 1 -- 172.21.15.27:0/2402978917 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc53c066f30 msgr2=0x7fc53c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:15.401 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.398+0000 7fc561535700 1 --2- 172.21.15.27:0/2402978917 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc53c066f30 0x7fc53c0693e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc55c159da0 tx=0x7fc544016000 comp rx=0 tx=0).stop 2023-10-06T01:42:15.401 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.398+0000 7fc561535700 1 -- 172.21.15.27:0/2402978917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc55c0ae860 msgr2=0x7fc55c0aa290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:15.401 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.398+0000 7fc561535700 1 --2- 172.21.15.27:0/2402978917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc55c0ae860 0x7fc55c0aa290 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fc55400cb10 tx=0x7fc554002820 comp rx=0 tx=0).stop 2023-10-06T01:42:15.401 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.398+0000 7fc561535700 1 -- 172.21.15.27:0/2402978917 shutdown_connections 2023-10-06T01:42:15.401 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.398+0000 7fc561535700 1 --2- 172.21.15.27:0/2402978917 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc53c066f30 0x7fc53c0693e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:15.401 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.398+0000 7fc561535700 1 --2- 172.21.15.27:0/2402978917 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc55c159950 0x7fc55c15bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:15.402 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.398+0000 7fc561535700 1 --2- 172.21.15.27:0/2402978917 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc55c0ae860 0x7fc55c0aa290 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:15.402 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.398+0000 7fc561535700 1 --2- 172.21.15.27:0/2402978917 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc55c159580 0x7fc55c153540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:15.402 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.398+0000 7fc561535700 1 -- 172.21.15.27:0/2402978917 >> 172.21.15.27:0/2402978917 conn(0x7fc55c06e6f0 msgr2=0x7fc55c14df70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:15.402 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.398+0000 7fc561535700 1 -- 172.21.15.27:0/2402978917 shutdown_connections 2023-10-06T01:42:15.402 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.398+0000 7fc561535700 1 -- 172.21.15.27:0/2402978917 wait complete. 2023-10-06T01:42:15.403 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 15 2023-10-06T01:42:15.412 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json flush journal 2023-10-06T01:42:15.561 INFO:teuthology.orchestra.run.smithi124.stdout:{"message":"","return_code":0} 2023-10-06T01:42:15.561 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2023-10-06T01:42:15.562 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001fb.00000000 parent 2023-10-06T01:42:15.601 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.600+0000 7fdfebc96000 1 Processor -- start 2023-10-06T01:42:15.601 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.600+0000 7fdfebc96000 1 -- start start 2023-10-06T01:42:15.603 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.600+0000 7fdfebc96000 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdfd00030b0 0x55a82570f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:15.604 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.600+0000 7fdfebc96000 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55a82570b530 0x55a825710ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:15.604 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.600+0000 7fdfebc96000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a825711410 0x55a825668a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:15.604 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.600+0000 7fdfebc96000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8255e5320 con 0x55a825711410 2023-10-06T01:42:15.604 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.601+0000 7fdfebc96000 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a825589980 con 0x7fdfd00030b0 2023-10-06T01:42:15.604 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.601+0000 7fdfebc96000 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a825589620 con 0x55a82570b530 2023-10-06T01:42:15.604 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.601+0000 7fdfe56d2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a825711410 0x55a825668a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:15.604 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.601+0000 7fdfe56d2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a825711410 0x55a825668a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59782/0 (socket says 172.21.15.27:59782) 2023-10-06T01:42:15.604 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.601+0000 7fdfe56d2700 1 -- 172.21.15.27:0/3048529665 learned_addr learned my addr 172.21.15.27:0/3048529665 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:15.604 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.601+0000 7fdfe4ed1700 1 --2- 172.21.15.27:0/3048529665 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdfd00030b0 0x55a82570f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:15.605 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.601+0000 7fdfe46d0700 1 --2- 172.21.15.27:0/3048529665 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55a82570b530 0x55a825710ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:15.605 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.601+0000 7fdfe56d2700 1 -- 172.21.15.27:0/3048529665 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55a82570b530 msgr2=0x55a825710ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:15.605 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.601+0000 7fdfe56d2700 1 --2- 172.21.15.27:0/3048529665 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55a82570b530 0x55a825710ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:15.605 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.601+0000 7fdfe56d2700 1 -- 172.21.15.27:0/3048529665 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdfd00030b0 msgr2=0x55a82570f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:15.605 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.601+0000 7fdfe56d2700 1 --2- 172.21.15.27:0/3048529665 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdfd00030b0 0x55a82570f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:15.605 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.601+0000 7fdfe56d2700 1 -- 172.21.15.27:0/3048529665 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a82558bae0 con 0x55a825711410 2023-10-06T01:42:15.605 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.601+0000 7fdfe56d2700 1 --2- 172.21.15.27:0/3048529665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a825711410 0x55a825668a80 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fdfd000ab90 tx=0x7fdfd4007040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ecb74176339f34f server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:15.605 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.601+0000 7fdfe36ce700 1 -- 172.21.15.27:0/3048529665 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd4013470 con 0x55a825711410 2023-10-06T01:42:15.605 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.601+0000 7fdfe36ce700 1 -- 172.21.15.27:0/3048529665 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfd40135d0 con 0x55a825711410 2023-10-06T01:42:15.605 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.601+0000 7fdfe36ce700 1 -- 172.21.15.27:0/3048529665 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd4013970 con 0x55a825711410 2023-10-06T01:42:15.605 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.602+0000 7fdfebc96000 1 -- 172.21.15.27:0/3048529665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a825711410 msgr2=0x55a825668a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:15.606 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.602+0000 7fdfebc96000 1 --2- 172.21.15.27:0/3048529665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a825711410 0x55a825668a80 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fdfd000ab90 tx=0x7fdfd4007040 comp rx=0 tx=0).stop 2023-10-06T01:42:15.606 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.602+0000 7fdfebc96000 1 -- 172.21.15.27:0/3048529665 shutdown_connections 2023-10-06T01:42:15.606 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.602+0000 7fdfebc96000 1 --2- 172.21.15.27:0/3048529665 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a825711410 0x55a825668a80 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:15.606 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.602+0000 7fdfebc96000 1 --2- 172.21.15.27:0/3048529665 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdfd00030b0 0x55a82570f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:15.606 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.602+0000 7fdfebc96000 1 --2- 172.21.15.27:0/3048529665 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55a82570b530 0x55a825710ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:15.606 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.602+0000 7fdfebc96000 1 -- 172.21.15.27:0/3048529665 >> 172.21.15.27:0/3048529665 conn(0x55a82563f8a0 msgr2=0x55a8256863f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:15.606 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.602+0000 7fdfebc96000 1 -- 172.21.15.27:0/3048529665 shutdown_connections 2023-10-06T01:42:15.606 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.602+0000 7fdfebc96000 1 -- 172.21.15.27:0/3048529665 wait complete. 2023-10-06T01:42:15.606 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.602+0000 7fdfebc96000 1 Processor -- start 2023-10-06T01:42:15.606 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.602+0000 7fdfebc96000 1 -- start start 2023-10-06T01:42:15.610 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.603+0000 7fdfebc96000 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55a825711410 0x55a825709b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:15.610 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.603+0000 7fdfebc96000 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdfd00030b0 0x55a8256842e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:15.610 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.603+0000 7fdfebc96000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a825684820 0x55a82570cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:15.610 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.603+0000 7fdfebc96000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a825580400 con 0x55a825684820 2023-10-06T01:42:15.610 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.603+0000 7fdfebc96000 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a825668f20 con 0x7fdfd00030b0 2023-10-06T01:42:15.610 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.603+0000 7fdfebc96000 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a8256691c0 con 0x55a825711410 2023-10-06T01:42:15.610 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.603+0000 7fdfe56d2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a825684820 0x55a82570cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:15.610 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.603+0000 7fdfe56d2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a825684820 0x55a82570cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59786/0 (socket says 172.21.15.27:59786) 2023-10-06T01:42:15.610 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.603+0000 7fdfe56d2700 1 -- 172.21.15.27:0/1023545998 learned_addr learned my addr 172.21.15.27:0/1023545998 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:15.611 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.603+0000 7fdfe46d0700 1 --2- 172.21.15.27:0/1023545998 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdfd00030b0 0x55a8256842e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:15.611 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.603+0000 7fdfe4ed1700 1 --2- 172.21.15.27:0/1023545998 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55a825711410 0x55a825709b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:15.611 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.603+0000 7fdfe56d2700 1 -- 172.21.15.27:0/1023545998 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55a825711410 msgr2=0x55a825709b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:15.611 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.603+0000 7fdfe56d2700 1 --2- 172.21.15.27:0/1023545998 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55a825711410 0x55a825709b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:15.611 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.603+0000 7fdfe56d2700 1 -- 172.21.15.27:0/1023545998 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdfd00030b0 msgr2=0x55a8256842e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:15.611 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.603+0000 7fdfe56d2700 1 --2- 172.21.15.27:0/1023545998 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdfd00030b0 0x55a8256842e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:15.611 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.603+0000 7fdfe56d2700 1 -- 172.21.15.27:0/1023545998 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd400a0e0 con 0x55a825684820 2023-10-06T01:42:15.611 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.603+0000 7fdfe46d0700 1 --2- 172.21.15.27:0/1023545998 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdfd00030b0 0x55a8256842e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:42:15.611 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.604+0000 7fdfe56d2700 1 --2- 172.21.15.27:0/1023545998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a825684820 0x55a82570cc00 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x55a82566ebc0 tx=0x7fdfd4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:15.611 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.604+0000 7fdfe1ecb700 1 -- 172.21.15.27:0/1023545998 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd4001090 con 0x55a825684820 2023-10-06T01:42:15.611 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.604+0000 7fdfe1ecb700 1 -- 172.21.15.27:0/1023545998 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfd4012a90 con 0x55a825684820 2023-10-06T01:42:15.612 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.604+0000 7fdfebc96000 1 -- 172.21.15.27:0/1023545998 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55a82558b540 con 0x55a825684820 2023-10-06T01:42:15.612 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.604+0000 7fdfe1ecb700 1 -- 172.21.15.27:0/1023545998 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfd4012e30 con 0x55a825684820 2023-10-06T01:42:15.612 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.604+0000 7fdfebc96000 1 -- 172.21.15.27:0/1023545998 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a825683e40 con 0x55a825684820 2023-10-06T01:42:15.612 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.605+0000 7fdfe1ecb700 1 -- 172.21.15.27:0/1023545998 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fdfd4025070 con 0x55a825684820 2023-10-06T01:42:15.612 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.605+0000 7fdfe1ecb700 1 --2- 172.21.15.27:0/1023545998 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdfc4065890 0x7fdfc4067d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:15.612 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.605+0000 7fdfe1ecb700 1 -- 172.21.15.27:0/1023545998 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fdfd40853a0 con 0x55a825684820 2023-10-06T01:42:15.612 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.606+0000 7fdfe4ed1700 1 --2- 172.21.15.27:0/1023545998 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdfc4065890 0x7fdfc4067d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:15.612 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.606+0000 7fdfebc96000 1 --2- 172.21.15.27:0/1023545998 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x55a825721790 0x55a82572a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:15.612 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.606+0000 7fdfebc96000 1 -- 172.21.15.27:0/1023545998 --> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] -- osd_op(unknown.0.0:1 6.13 6:ca8fdbba:::100000001fb.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31) v8 -- 0x55a825721b90 con 0x55a825721790 2023-10-06T01:42:15.612 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.606+0000 7fdfe46d0700 1 --2- 172.21.15.27:0/1023545998 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x55a825721790 0x55a82572a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:15.612 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.606+0000 7fdfe4ed1700 1 --2- 172.21.15.27:0/1023545998 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdfc4065890 0x7fdfc4067d40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc009aa0 tx=0x7fdfdc011040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:15.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.606+0000 7fdfe46d0700 1 --2- 172.21.15.27:0/1023545998 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x55a825721790 0x55a82572a250 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:15.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.606+0000 7fdfe46d0700 1 -- 172.21.15.27:0/1023545998 <== osd.3 v2:172.21.15.27:6824/90668 1 ==== osd_op_reply(1 100000001fb.00000000 [getxattr (95) out=95b] v0'0 uv2 ondisk = 0) v8 ==== 164+0+95 (crc 0 0 0) 0x7fdfcc016070 con 0x55a825721790 2023-10-06T01:42:15.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.606+0000 7fdfebc96000 1 -- 172.21.15.27:0/1023545998 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x55a825721790 msgr2=0x55a82572a250 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:15.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.606+0000 7fdfebc96000 1 --2- 172.21.15.27:0/1023545998 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x55a825721790 0x55a82572a250 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:15.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.606+0000 7fdfebc96000 1 -- 172.21.15.27:0/1023545998 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdfc4065890 msgr2=0x7fdfc4067d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:15.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.606+0000 7fdfebc96000 1 --2- 172.21.15.27:0/1023545998 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdfc4065890 0x7fdfc4067d40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc009aa0 tx=0x7fdfdc011040 comp rx=0 tx=0).stop 2023-10-06T01:42:15.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.606+0000 7fdfebc96000 1 -- 172.21.15.27:0/1023545998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a825684820 msgr2=0x55a82570cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:15.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.606+0000 7fdfebc96000 1 --2- 172.21.15.27:0/1023545998 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55a825684820 0x55a82570cc00 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x55a82566ebc0 tx=0x7fdfd4019000 comp rx=0 tx=0).stop 2023-10-06T01:42:15.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.606+0000 7fdfe56d2700 1 -- 172.21.15.27:0/1023545998 reap_dead start 2023-10-06T01:42:15.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.606+0000 7fdfebc96000 1 -- 172.21.15.27:0/1023545998 shutdown_connections 2023-10-06T01:42:15.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.606+0000 7fdfebc96000 1 -- 172.21.15.27:0/1023545998 >> 172.21.15.27:0/1023545998 conn(0x55a82563f8a0 msgr2=0x55a8256863f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:15.613 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.607+0000 7fdfebc96000 1 -- 172.21.15.27:0/1023545998 shutdown_connections 2023-10-06T01:42:15.614 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.607+0000 7fdfebc96000 1 -- 172.21.15.27:0/1023545998 wait complete. 2023-10-06T01:42:15.614 INFO:teuthology.orchestra.run.smithi027.stdout:Y��alphaparent_b 2023-10-06T01:42:15.615 DEBUG:teuthology.orchestra.run.smithi027:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2023-10-06T01:42:15.707 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2023-10-06T01:42:15.708 INFO:teuthology.orchestra.run.smithi027.stdout: "ino": 1099511628283, 2023-10-06T01:42:15.708 INFO:teuthology.orchestra.run.smithi027.stdout: "ancestors": [ 2023-10-06T01:42:15.708 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:42:15.708 INFO:teuthology.orchestra.run.smithi027.stdout: "dirino": 1099511628284, 2023-10-06T01:42:15.708 INFO:teuthology.orchestra.run.smithi027.stdout: "dname": "alpha", 2023-10-06T01:42:15.708 INFO:teuthology.orchestra.run.smithi027.stdout: "version": 3 2023-10-06T01:42:15.708 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2023-10-06T01:42:15.708 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:42:15.708 INFO:teuthology.orchestra.run.smithi027.stdout: "dirino": 1, 2023-10-06T01:42:15.708 INFO:teuthology.orchestra.run.smithi027.stdout: "dname": "parent_b", 2023-10-06T01:42:15.708 INFO:teuthology.orchestra.run.smithi027.stdout: "version": 25 2023-10-06T01:42:15.708 INFO:teuthology.orchestra.run.smithi027.stdout: } 2023-10-06T01:42:15.709 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2023-10-06T01:42:15.709 INFO:teuthology.orchestra.run.smithi027.stdout: "pool": 6, 2023-10-06T01:42:15.709 INFO:teuthology.orchestra.run.smithi027.stdout: "old_pools": [] 2023-10-06T01:42:15.709 INFO:teuthology.orchestra.run.smithi027.stdout:} 2023-10-06T01:42:15.709 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:15.710 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create data_new --pg_num_min 64 2023-10-06T01:42:15.872 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.871+0000 7fb14c35d700 1 Processor -- start 2023-10-06T01:42:15.873 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.871+0000 7fb14c35d700 1 -- start start 2023-10-06T01:42:15.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.872+0000 7fb14c35d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb1440ccbe0 0x7fb1440b5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:15.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.872+0000 7fb14c35d700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1440cbed0 con 0x7fb1440d0f00 2023-10-06T01:42:15.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.872+0000 7fb14c35d700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1440cc750 con 0x7fb1440b55b0 2023-10-06T01:42:15.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.872+0000 7fb14c35d700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb144159d40 con 0x7fb1440ccbe0 2023-10-06T01:42:15.876 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.872+0000 7fb14a8fa700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb1440d0f00 0x7fb1440ce270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40344/0 (socket says 172.21.15.27:40344) 2023-10-06T01:42:15.876 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.872+0000 7fb14a8fa700 1 -- 172.21.15.27:0/3294966335 learned_addr learned my addr 172.21.15.27:0/3294966335 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:15.876 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.872+0000 7fb1498f8700 1 --2- 172.21.15.27:0/3294966335 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb1440ccbe0 0x7fb1440b5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:15.876 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.873+0000 7fb1490f7700 1 -- 172.21.15.27:0/3294966335 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2689223130 0 0) 0x7fb1440cbed0 con 0x7fb1440d0f00 2023-10-06T01:42:15.876 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.873+0000 7fb1490f7700 1 -- 172.21.15.27:0/3294966335 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb12c003570 con 0x7fb1440d0f00 2023-10-06T01:42:15.876 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.873+0000 7fb1490f7700 1 -- 172.21.15.27:0/3294966335 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4129643978 0 0) 0x7fb1440cc750 con 0x7fb1440b55b0 2023-10-06T01:42:15.876 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.873+0000 7fb1490f7700 1 -- 172.21.15.27:0/3294966335 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1440cbed0 con 0x7fb1440b55b0 2023-10-06T01:42:15.876 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.873+0000 7fb1498f8700 1 -- 172.21.15.27:0/3294966335 >> v1:172.21.15.124:6789/0 conn(0x7fb1440b55b0 legacy=0x7fb1440cf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:15.876 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.873+0000 7fb1498f8700 1 -- 172.21.15.27:0/3294966335 >> v1:172.21.15.27:6789/0 conn(0x7fb1440d0f00 legacy=0x7fb1440ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:15.876 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.873+0000 7fb1498f8700 1 -- 172.21.15.27:0/3294966335 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1440cd050 con 0x7fb1440ccbe0 2023-10-06T01:42:15.876 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.873+0000 7fb1498f8700 1 --2- 172.21.15.27:0/3294966335 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb1440ccbe0 0x7fb1440b5980 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb138009910 tx=0x7fb1380091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=952252632c171fc8 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:15.877 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.873+0000 7fb1490f7700 1 -- 172.21.15.27:0/3294966335 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb138011630 con 0x7fb1440ccbe0 2023-10-06T01:42:15.877 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.873+0000 7fb1490f7700 1 -- 172.21.15.27:0/3294966335 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb138011790 con 0x7fb1440ccbe0 2023-10-06T01:42:15.877 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.873+0000 7fb1490f7700 1 -- 172.21.15.27:0/3294966335 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb138011b30 con 0x7fb1440ccbe0 2023-10-06T01:42:15.877 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.874+0000 7fb14c35d700 1 -- 172.21.15.27:0/3294966335 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb1440ccbe0 msgr2=0x7fb1440b5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:15.877 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.874+0000 7fb14c35d700 1 --2- 172.21.15.27:0/3294966335 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb1440ccbe0 0x7fb1440b5980 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb138009910 tx=0x7fb1380091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:15.877 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.874+0000 7fb14c35d700 1 -- 172.21.15.27:0/3294966335 shutdown_connections 2023-10-06T01:42:15.877 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.874+0000 7fb14c35d700 1 --2- 172.21.15.27:0/3294966335 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb1440ccbe0 0x7fb1440b5980 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:15.877 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.874+0000 7fb14c35d700 1 -- 172.21.15.27:0/3294966335 >> 172.21.15.27:0/3294966335 conn(0x7fb14408a050 msgr2=0x7fb144099970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:15.877 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.874+0000 7fb14c35d700 1 -- 172.21.15.27:0/3294966335 shutdown_connections 2023-10-06T01:42:15.877 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.874+0000 7fb14c35d700 1 -- 172.21.15.27:0/3294966335 wait complete. 2023-10-06T01:42:15.880 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.875+0000 7fb14c35d700 1 Processor -- start 2023-10-06T01:42:15.880 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.875+0000 7fb14c35d700 1 -- start start 2023-10-06T01:42:15.880 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.876+0000 7fb14c35d700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb1440b55b0 0x7fb144156c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:15.880 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.876+0000 7fb14c35d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb1440ccbe0 0x7fb144157950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:15.880 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.876+0000 7fb14c35d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1440d0f00 0x7fb144157e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:15.880 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.876+0000 7fb14c35d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb144169910 con 0x7fb1440d0f00 2023-10-06T01:42:15.880 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.876+0000 7fb14c35d700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb144169bf0 con 0x7fb1440b55b0 2023-10-06T01:42:15.881 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.876+0000 7fb14c35d700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb144169ed0 con 0x7fb1440ccbe0 2023-10-06T01:42:15.881 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.876+0000 7fb14a8fa700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1440d0f00 0x7fb144157e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:15.881 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.876+0000 7fb14a8fa700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1440d0f00 0x7fb144157e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59788/0 (socket says 172.21.15.27:59788) 2023-10-06T01:42:15.881 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.876+0000 7fb14a8fa700 1 -- 172.21.15.27:0/1435060552 learned_addr learned my addr 172.21.15.27:0/1435060552 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:15.881 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.876+0000 7fb14a0f9700 1 --2- 172.21.15.27:0/1435060552 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb1440b55b0 0x7fb144156c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:15.881 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.876+0000 7fb1498f8700 1 --2- 172.21.15.27:0/1435060552 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb1440ccbe0 0x7fb144157950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:15.881 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.876+0000 7fb14a8fa700 1 -- 172.21.15.27:0/1435060552 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb1440ccbe0 msgr2=0x7fb144157950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:15.881 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.876+0000 7fb14a8fa700 1 --2- 172.21.15.27:0/1435060552 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb1440ccbe0 0x7fb144157950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:15.881 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.876+0000 7fb14a8fa700 1 -- 172.21.15.27:0/1435060552 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb1440b55b0 msgr2=0x7fb144156c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:15.881 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.876+0000 7fb14a8fa700 1 --2- 172.21.15.27:0/1435060552 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb1440b55b0 0x7fb144156c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:15.882 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.876+0000 7fb14a8fa700 1 -- 172.21.15.27:0/1435060552 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb138009420 con 0x7fb1440d0f00 2023-10-06T01:42:15.882 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.876+0000 7fb1498f8700 1 --2- 172.21.15.27:0/1435060552 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb1440ccbe0 0x7fb144157950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:42:15.882 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.876+0000 7fb14a8fa700 1 --2- 172.21.15.27:0/1435060552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1440d0f00 0x7fb144157e90 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb13400c560 tx=0x7fb134007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:15.882 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.876+0000 7fb1427fc700 1 -- 172.21.15.27:0/1435060552 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb134005c00 con 0x7fb1440d0f00 2023-10-06T01:42:15.882 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.876+0000 7fb1427fc700 1 -- 172.21.15.27:0/1435060552 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb134005d80 con 0x7fb1440d0f00 2023-10-06T01:42:15.882 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.876+0000 7fb14c35d700 1 -- 172.21.15.27:0/1435060552 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb14416a1b0 con 0x7fb1440d0f00 2023-10-06T01:42:15.882 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.876+0000 7fb1427fc700 1 -- 172.21.15.27:0/1435060552 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb134007320 con 0x7fb1440d0f00 2023-10-06T01:42:15.882 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.877+0000 7fb14c35d700 1 -- 172.21.15.27:0/1435060552 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1440683f0 con 0x7fb1440d0f00 2023-10-06T01:42:15.882 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.877+0000 7fb14c35d700 1 -- 172.21.15.27:0/1435060552 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb14416a500 con 0x7fb1440d0f00 2023-10-06T01:42:15.882 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.878+0000 7fb1427fc700 1 -- 172.21.15.27:0/1435060552 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb134007690 con 0x7fb1440d0f00 2023-10-06T01:42:15.882 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.878+0000 7fb1427fc700 1 --2- 172.21.15.27:0/1435060552 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb130066f30 0x7fb1300693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:15.883 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.878+0000 7fb1427fc700 1 -- 172.21.15.27:0/1435060552 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fb134083b10 con 0x7fb1440d0f00 2023-10-06T01:42:15.883 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.878+0000 7fb14a0f9700 1 --2- 172.21.15.27:0/1435060552 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb130066f30 0x7fb1300693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:15.883 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.879+0000 7fb14a0f9700 1 --2- 172.21.15.27:0/1435060552 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb130066f30 0x7fb1300693e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb14415f320 tx=0x7fb13c002b70 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:15.884 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:15.883+0000 7fb1427fc700 1 -- 172.21.15.27:0/1435060552 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb134050c20 con 0x7fb1440d0f00 2023-10-06T01:42:16.075 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.074+0000 7fb14c35d700 1 -- 172.21.15.27:0/1435060552 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "data_new", "pg_num_min": 64} v 0) v1 -- 0x7fb1440717c0 con 0x7fb1440d0f00 2023-10-06T01:42:16.365 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.364+0000 7fb1427fc700 1 -- 172.21.15.27:0/1435060552 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "data_new", "pg_num_min": 64}]=0 pool 'data_new' created v32) v1 ==== 124+0+0 (secure 0 0 0) 0x7fb134017070 con 0x7fb1440d0f00 2023-10-06T01:42:16.368 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.367+0000 7fb14c35d700 1 -- 172.21.15.27:0/1435060552 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb130066f30 msgr2=0x7fb1300693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:16.368 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.367+0000 7fb14c35d700 1 --2- 172.21.15.27:0/1435060552 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb130066f30 0x7fb1300693e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb14415f320 tx=0x7fb13c002b70 comp rx=0 tx=0).stop 2023-10-06T01:42:16.369 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.367+0000 7fb14c35d700 1 -- 172.21.15.27:0/1435060552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1440d0f00 msgr2=0x7fb144157e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:16.369 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.367+0000 7fb14c35d700 1 --2- 172.21.15.27:0/1435060552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1440d0f00 0x7fb144157e90 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb13400c560 tx=0x7fb134007dd0 comp rx=0 tx=0).stop 2023-10-06T01:42:16.369 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.367+0000 7fb14c35d700 1 -- 172.21.15.27:0/1435060552 shutdown_connections 2023-10-06T01:42:16.369 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.367+0000 7fb14c35d700 1 --2- 172.21.15.27:0/1435060552 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb130066f30 0x7fb1300693e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:16.369 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.367+0000 7fb14c35d700 1 --2- 172.21.15.27:0/1435060552 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb1440d0f00 0x7fb144157e90 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:16.369 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.367+0000 7fb14c35d700 1 --2- 172.21.15.27:0/1435060552 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb1440b55b0 0x7fb144156c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:16.369 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.367+0000 7fb14c35d700 1 --2- 172.21.15.27:0/1435060552 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb1440ccbe0 0x7fb144157950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:16.369 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.367+0000 7fb14c35d700 1 -- 172.21.15.27:0/1435060552 >> 172.21.15.27:0/1435060552 conn(0x7fb14408a050 msgr2=0x7fb144159060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:16.369 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.367+0000 7fb14c35d700 1 -- 172.21.15.27:0/1435060552 shutdown_connections 2023-10-06T01:42:16.369 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.367+0000 7fb14c35d700 1 -- 172.21.15.27:0/1435060552 wait complete. 2023-10-06T01:42:16.374 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'data_new' created 2023-10-06T01:42:16.384 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs data_new 2023-10-06T01:42:16.504 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.503+0000 7fdf9e875700 1 Processor -- start 2023-10-06T01:42:16.504 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.503+0000 7fdf9e875700 1 -- start start 2023-10-06T01:42:16.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.504+0000 7fdf9e875700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdf98142e90 0x7fdf98069810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:16.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.504+0000 7fdf9e875700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fdf98069d50 0x7fdf9815d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:16.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.504+0000 7fdf9e875700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf98069380 con 0x7fdf9815de80 2023-10-06T01:42:16.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.504+0000 7fdf9e875700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf98068770 con 0x7fdf98069d50 2023-10-06T01:42:16.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.504+0000 7fdf9e875700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf98068a50 con 0x7fdf98142e90 2023-10-06T01:42:16.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.504+0000 7fdf9e074700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fdf9815de80 0x7fdf980b0370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40346/0 (socket says 172.21.15.27:40346) 2023-10-06T01:42:16.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.504+0000 7fdf9e074700 1 -- 172.21.15.27:0/3700235402 learned_addr learned my addr 172.21.15.27:0/3700235402 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:16.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.504+0000 7fdf9d072700 1 --2- 172.21.15.27:0/3700235402 >> v2:172.21.15.124:3300/0 conn(0x7fdf98069d50 0x7fdf9815d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:16.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.504+0000 7fdf9d873700 1 --2- 172.21.15.27:0/3700235402 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdf98142e90 0x7fdf98069810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:16.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.504+0000 7fdf9c871700 1 -- 172.21.15.27:0/3700235402 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3668653351 0 0) 0x7fdf98069380 con 0x7fdf9815de80 2023-10-06T01:42:16.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.504+0000 7fdf9c871700 1 -- 172.21.15.27:0/3700235402 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf80003570 con 0x7fdf9815de80 2023-10-06T01:42:16.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.505+0000 7fdf9c871700 1 -- 172.21.15.27:0/3700235402 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2004515154 0 0) 0x7fdf80003570 con 0x7fdf9815de80 2023-10-06T01:42:16.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.505+0000 7fdf9c871700 1 -- 172.21.15.27:0/3700235402 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdf98142e90 msgr2=0x7fdf98069810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:16.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.505+0000 7fdf9c871700 1 --2- 172.21.15.27:0/3700235402 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdf98142e90 0x7fdf98069810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:16.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.505+0000 7fdf9c871700 1 -- 172.21.15.27:0/3700235402 >> v2:172.21.15.124:3300/0 conn(0x7fdf98069d50 msgr2=0x7fdf9815d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:16.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.505+0000 7fdf9c871700 1 --2- 172.21.15.27:0/3700235402 >> v2:172.21.15.124:3300/0 conn(0x7fdf98069d50 0x7fdf9815d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:16.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.505+0000 7fdf9c871700 1 -- 172.21.15.27:0/3700235402 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf98068d30 con 0x7fdf9815de80 2023-10-06T01:42:16.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.505+0000 7fdf9c871700 1 -- 172.21.15.27:0/3700235402 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2619078944 0 0) 0x7fdf88001470 con 0x7fdf9815de80 2023-10-06T01:42:16.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.505+0000 7fdf9c871700 1 -- 172.21.15.27:0/3700235402 >> v1:172.21.15.27:6789/0 conn(0x7fdf9815de80 legacy=0x7fdf980b0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:16.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.505+0000 7fdf9c871700 1 --2- 172.21.15.27:0/3700235402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf80003a60 0x7fdf80003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:16.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.505+0000 7fdf9c871700 1 --2- 172.21.15.27:0/3700235402 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdf98142e90 0x7fdf80004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:16.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.505+0000 7fdf9c871700 1 --2- 172.21.15.27:0/3700235402 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdf80004d50 0x7fdf800051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:16.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.505+0000 7fdf9c871700 1 -- 172.21.15.27:0/3700235402 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf98068770 con 0x7fdf80003a60 2023-10-06T01:42:16.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.505+0000 7fdf9c871700 1 -- 172.21.15.27:0/3700235402 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf98068a50 con 0x7fdf80004d50 2023-10-06T01:42:16.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.505+0000 7fdf9c871700 1 -- 172.21.15.27:0/3700235402 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8000ba60 con 0x7fdf98142e90 2023-10-06T01:42:16.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.505+0000 7fdf9d873700 1 --2- 172.21.15.27:0/3700235402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf80003a60 0x7fdf80003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:16.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.505+0000 7fdf9d072700 1 --2- 172.21.15.27:0/3700235402 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdf98142e90 0x7fdf80004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:16.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.505+0000 7fdf9e074700 1 --2- 172.21.15.27:0/3700235402 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdf80004d50 0x7fdf800051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:16.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.505+0000 7fdf9d873700 1 -- 172.21.15.27:0/3700235402 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdf98142e90 msgr2=0x7fdf80004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:16.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.505+0000 7fdf9d873700 1 --2- 172.21.15.27:0/3700235402 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdf98142e90 0x7fdf80004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:16.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.505+0000 7fdf9d873700 1 -- 172.21.15.27:0/3700235402 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdf80004d50 msgr2=0x7fdf800051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:16.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.505+0000 7fdf9d873700 1 --2- 172.21.15.27:0/3700235402 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdf80004d50 0x7fdf800051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:16.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.505+0000 7fdf9d873700 1 -- 172.21.15.27:0/3700235402 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf80003570 con 0x7fdf80003a60 2023-10-06T01:42:16.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.505+0000 7fdf9d873700 1 --2- 172.21.15.27:0/3700235402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf80003a60 0x7fdf80003e70 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fdf9000e210 tx=0x7fdf9000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69d4ddd2850c8b26 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:16.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.505+0000 7fdf9c871700 1 -- 172.21.15.27:0/3700235402 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf9000ea60 con 0x7fdf80003a60 2023-10-06T01:42:16.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.505+0000 7fdf9c871700 1 -- 172.21.15.27:0/3700235402 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf9000f040 con 0x7fdf80003a60 2023-10-06T01:42:16.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.506+0000 7fdf9c871700 1 -- 172.21.15.27:0/3700235402 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf9000ee10 con 0x7fdf80003a60 2023-10-06T01:42:16.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.506+0000 7fdf9e875700 1 -- 172.21.15.27:0/3700235402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf80003a60 msgr2=0x7fdf80003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:16.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.506+0000 7fdf9e875700 1 --2- 172.21.15.27:0/3700235402 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf80003a60 0x7fdf80003e70 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fdf9000e210 tx=0x7fdf9000b040 comp rx=0 tx=0).stop 2023-10-06T01:42:16.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.506+0000 7fdf9e074700 1 -- 172.21.15.27:0/3700235402 reap_dead start 2023-10-06T01:42:16.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.506+0000 7fdf9e875700 1 -- 172.21.15.27:0/3700235402 shutdown_connections 2023-10-06T01:42:16.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.506+0000 7fdf9e875700 1 -- 172.21.15.27:0/3700235402 >> 172.21.15.27:0/3700235402 conn(0x7fdf980ccfe0 msgr2=0x7fdf980cd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:16.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.506+0000 7fdf9e875700 1 -- 172.21.15.27:0/3700235402 shutdown_connections 2023-10-06T01:42:16.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.506+0000 7fdf9e875700 1 -- 172.21.15.27:0/3700235402 wait complete. 2023-10-06T01:42:16.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.507+0000 7fdf9e875700 1 Processor -- start 2023-10-06T01:42:16.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.507+0000 7fdf9e875700 1 -- start start 2023-10-06T01:42:16.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.508+0000 7fdf9e875700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf80003a60 0x7fdf980c5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:16.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.508+0000 7fdf9e875700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdf980c85f0 0x7fdf98158d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:16.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.508+0000 7fdf9e875700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdf980c9f00 0x7fdf980b4c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:16.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.508+0000 7fdf9e875700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf981592c0 con 0x7fdf80003a60 2023-10-06T01:42:16.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.508+0000 7fdf9e875700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf980c6620 con 0x7fdf980c85f0 2023-10-06T01:42:16.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.508+0000 7fdf9e875700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf980c8a60 con 0x7fdf980c9f00 2023-10-06T01:42:16.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.508+0000 7fdf9d873700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf80003a60 0x7fdf980c5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:16.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.508+0000 7fdf9d873700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf80003a60 0x7fdf980c5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59812/0 (socket says 172.21.15.27:59812) 2023-10-06T01:42:16.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.508+0000 7fdf9d873700 1 -- 172.21.15.27:0/3895577767 learned_addr learned my addr 172.21.15.27:0/3895577767 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:16.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.508+0000 7fdf9e074700 1 --2- 172.21.15.27:0/3895577767 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdf980c9f00 0x7fdf980b4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:16.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.508+0000 7fdf9d873700 1 -- 172.21.15.27:0/3895577767 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdf980c9f00 msgr2=0x7fdf980b4c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:16.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.508+0000 7fdf9d873700 1 --2- 172.21.15.27:0/3895577767 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdf980c9f00 0x7fdf980b4c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:16.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.508+0000 7fdf9d072700 1 --2- 172.21.15.27:0/3895577767 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdf980c85f0 0x7fdf98158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:16.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.508+0000 7fdf9d873700 1 -- 172.21.15.27:0/3895577767 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdf980c85f0 msgr2=0x7fdf98158d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:16.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.508+0000 7fdf9d873700 1 --2- 172.21.15.27:0/3895577767 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdf980c85f0 0x7fdf98158d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:16.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.508+0000 7fdf9d873700 1 -- 172.21.15.27:0/3895577767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf980c5a10 con 0x7fdf80003a60 2023-10-06T01:42:16.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.509+0000 7fdf9d873700 1 --2- 172.21.15.27:0/3895577767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf80003a60 0x7fdf980c5310 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fdf80004730 tx=0x7fdf9001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:16.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.509+0000 7fdf95ffb700 1 -- 172.21.15.27:0/3895577767 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf9001d420 con 0x7fdf80003a60 2023-10-06T01:42:16.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.509+0000 7fdf95ffb700 1 -- 172.21.15.27:0/3895577767 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf9001d580 con 0x7fdf80003a60 2023-10-06T01:42:16.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.509+0000 7fdf95ffb700 1 -- 172.21.15.27:0/3895577767 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf9001d940 con 0x7fdf80003a60 2023-10-06T01:42:16.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.509+0000 7fdf9e875700 1 -- 172.21.15.27:0/3895577767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf980c5cf0 con 0x7fdf80003a60 2023-10-06T01:42:16.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.509+0000 7fdf9e875700 1 -- 172.21.15.27:0/3895577767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf980c62b0 con 0x7fdf80003a60 2023-10-06T01:42:16.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.510+0000 7fdf9e875700 1 -- 172.21.15.27:0/3895577767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf9804c3c0 con 0x7fdf80003a60 2023-10-06T01:42:16.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.511+0000 7fdf95ffb700 1 -- 172.21.15.27:0/3895577767 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fdf90003040 con 0x7fdf80003a60 2023-10-06T01:42:16.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.511+0000 7fdf95ffb700 1 --2- 172.21.15.27:0/3895577767 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdf84066f30 0x7fdf840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:16.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.511+0000 7fdf95ffb700 1 -- 172.21.15.27:0/3895577767 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6421+0+0 (secure 0 0 0) 0x7fdf90086c70 con 0x7fdf80003a60 2023-10-06T01:42:16.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.511+0000 7fdf9d072700 1 --2- 172.21.15.27:0/3895577767 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdf84066f30 0x7fdf840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:16.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.512+0000 7fdf9d072700 1 --2- 172.21.15.27:0/3895577767 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdf84066f30 0x7fdf840693e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c0093f0 tx=0x7fdf8c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:16.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.516+0000 7fdf95ffb700 1 -- 172.21.15.27:0/3895577767 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdf90053c50 con 0x7fdf80003a60 2023-10-06T01:42:16.756 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:16.755+0000 7fdf9e875700 1 -- 172.21.15.27:0/3895577767 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "data_new"} v 0) v1 -- 0x7fdf98069010 con 0x7fdf80003a60 2023-10-06T01:42:17.369 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.367+0000 7fdf95ffb700 1 -- 172.21.15.27:0/3895577767 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "data_new"}]=0 added data pool 7 to fsmap v16) v1 ==== 131+0+0 (secure 0 0 0) 0x7fdf90018020 con 0x7fdf80003a60 2023-10-06T01:42:17.377 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.376+0000 7fdf9e875700 1 -- 172.21.15.27:0/3895577767 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdf84066f30 msgr2=0x7fdf840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:17.377 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.376+0000 7fdf9e875700 1 --2- 172.21.15.27:0/3895577767 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdf84066f30 0x7fdf840693e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c0093f0 tx=0x7fdf8c008040 comp rx=0 tx=0).stop 2023-10-06T01:42:17.377 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.376+0000 7fdf9e875700 1 -- 172.21.15.27:0/3895577767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf80003a60 msgr2=0x7fdf980c5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:17.377 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.376+0000 7fdf9e875700 1 --2- 172.21.15.27:0/3895577767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf80003a60 0x7fdf980c5310 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fdf80004730 tx=0x7fdf9001a000 comp rx=0 tx=0).stop 2023-10-06T01:42:17.381 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.380+0000 7fdf9e875700 1 -- 172.21.15.27:0/3895577767 shutdown_connections 2023-10-06T01:42:17.382 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.380+0000 7fdf9e875700 1 --2- 172.21.15.27:0/3895577767 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fdf84066f30 0x7fdf840693e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:17.382 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.380+0000 7fdf9e875700 1 --2- 172.21.15.27:0/3895577767 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fdf980c9f00 0x7fdf980b4c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:17.382 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.380+0000 7fdf9e875700 1 --2- 172.21.15.27:0/3895577767 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fdf80003a60 0x7fdf980c5310 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:17.382 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.380+0000 7fdf9e875700 1 --2- 172.21.15.27:0/3895577767 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fdf980c85f0 0x7fdf98158d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:17.382 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.380+0000 7fdf9e875700 1 -- 172.21.15.27:0/3895577767 >> 172.21.15.27:0/3895577767 conn(0x7fdf980ccfe0 msgr2=0x7fdf9815d220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:17.382 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.380+0000 7fdf9e875700 1 -- 172.21.15.27:0/3895577767 shutdown_connections 2023-10-06T01:42:17.382 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.380+0000 7fdf9e875700 1 -- 172.21.15.27:0/3895577767 wait complete. 2023-10-06T01:42:17.382 INFO:teuthology.orchestra.run.smithi027.stderr:added data pool 7 to fsmap 2023-10-06T01:42:17.395 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T01:42:17.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.515+0000 7f09fed1c700 1 Processor -- start 2023-10-06T01:42:17.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.515+0000 7f09fed1c700 1 -- start start 2023-10-06T01:42:17.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.516+0000 7f09fed1c700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f09f815d850 0x7f09f80b5c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:17.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.516+0000 7f09fed1c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f09f80b6170 0x7f09f80b0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:17.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.516+0000 7f09fed1c700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09f8069f70 con 0x7f09f815d480 2023-10-06T01:42:17.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.516+0000 7f09fed1c700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09f80687c0 con 0x7f09f80b6170 2023-10-06T01:42:17.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.516+0000 7f09fed1c700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09f8068aa0 con 0x7f09f815d850 2023-10-06T01:42:17.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.516+0000 7f09fe51b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f09f80b6170 0x7f09f80b0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:17.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.516+0000 7f09fe51b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f09f80b6170 0x7f09f80b0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59816/0 (socket says 172.21.15.27:59816) 2023-10-06T01:42:17.519 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.516+0000 7f09fe51b700 1 -- 172.21.15.27:0/1822591190 learned_addr learned my addr 172.21.15.27:0/1822591190 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:17.520 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.516+0000 7f09fd519700 1 --2- 172.21.15.27:0/1822591190 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f09f815d850 0x7f09f80b5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:17.520 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.516+0000 7f09fe51b700 1 -- 172.21.15.27:0/1822591190 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f09f815d850 msgr2=0x7f09f80b5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:17.520 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.516+0000 7f09fe51b700 1 --2- 172.21.15.27:0/1822591190 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f09f815d850 0x7f09f80b5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:17.520 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.516+0000 7f09fe51b700 1 -- 172.21.15.27:0/1822591190 >> v1:172.21.15.124:6789/0 conn(0x7f09f815d480 legacy=0x7f09f8069860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:17.520 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.516+0000 7f09fe51b700 1 -- 172.21.15.27:0/1822591190 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f8068d80 con 0x7f09f80b6170 2023-10-06T01:42:17.520 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.517+0000 7f09fe51b700 1 --2- 172.21.15.27:0/1822591190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09f80b6170 0x7f09f80b0310 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f09e8009f20 tx=0x7f09e8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9810d4671190824c server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:17.520 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.517+0000 7f09fcd18700 1 -- 172.21.15.27:0/1822591190 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e800e070 con 0x7f09f80b6170 2023-10-06T01:42:17.520 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.517+0000 7f09fcd18700 1 -- 172.21.15.27:0/1822591190 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09e800ba80 con 0x7f09f80b6170 2023-10-06T01:42:17.520 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.517+0000 7f09fcd18700 1 -- 172.21.15.27:0/1822591190 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e80092e0 con 0x7f09f80b6170 2023-10-06T01:42:17.520 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.517+0000 7f09fed1c700 1 -- 172.21.15.27:0/1822591190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09f80b6170 msgr2=0x7f09f80b0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:17.520 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.517+0000 7f09fed1c700 1 --2- 172.21.15.27:0/1822591190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09f80b6170 0x7f09f80b0310 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f09e8009f20 tx=0x7f09e8009840 comp rx=0 tx=0).stop 2023-10-06T01:42:17.520 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.517+0000 7f09fed1c700 1 -- 172.21.15.27:0/1822591190 shutdown_connections 2023-10-06T01:42:17.521 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.517+0000 7f09fed1c700 1 --2- 172.21.15.27:0/1822591190 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09f80b6170 0x7f09f80b0310 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:17.521 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.517+0000 7f09fed1c700 1 --2- 172.21.15.27:0/1822591190 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f09f815d850 0x7f09f80b5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:17.521 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.517+0000 7f09fed1c700 1 -- 172.21.15.27:0/1822591190 >> 172.21.15.27:0/1822591190 conn(0x7f09f80ccf80 msgr2=0x7f09f80cd380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:17.521 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.517+0000 7f09fed1c700 1 -- 172.21.15.27:0/1822591190 shutdown_connections 2023-10-06T01:42:17.521 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.518+0000 7f09fed1c700 1 -- 172.21.15.27:0/1822591190 wait complete. 2023-10-06T01:42:17.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.519+0000 7f09fed1c700 1 Processor -- start 2023-10-06T01:42:17.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.519+0000 7f09fed1c700 1 -- start start 2023-10-06T01:42:17.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.519+0000 7f09fed1c700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f09f80b6170 0x7f09f816cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:17.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.519+0000 7f09fed1c700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f09f815d480 0x7f09f812f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:17.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.519+0000 7f09fed1c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09f815d850 0x7f09f812fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:17.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.519+0000 7f09fed1c700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f8140b40 con 0x7f09f815d850 2023-10-06T01:42:17.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.519+0000 7f09fed1c700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f8130e40 con 0x7f09f80b6170 2023-10-06T01:42:17.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.519+0000 7f09fed1c700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09f8130230 con 0x7f09f815d480 2023-10-06T01:42:17.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.519+0000 7f09fe51b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09f815d850 0x7f09f812fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:17.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.520+0000 7f09fe51b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09f815d850 0x7f09f812fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59820/0 (socket says 172.21.15.27:59820) 2023-10-06T01:42:17.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.520+0000 7f09fe51b700 1 -- 172.21.15.27:0/3851996826 learned_addr learned my addr 172.21.15.27:0/3851996826 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:17.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.520+0000 7f09fdd1a700 1 --2- 172.21.15.27:0/3851996826 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f09f80b6170 0x7f09f816cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:17.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.520+0000 7f09fd519700 1 --2- 172.21.15.27:0/3851996826 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f09f815d480 0x7f09f812f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:17.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.520+0000 7f09fe51b700 1 -- 172.21.15.27:0/3851996826 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f09f815d480 msgr2=0x7f09f812f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:17.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.520+0000 7f09fe51b700 1 --2- 172.21.15.27:0/3851996826 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f09f815d480 0x7f09f812f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:17.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.520+0000 7f09fe51b700 1 -- 172.21.15.27:0/3851996826 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f09f80b6170 msgr2=0x7f09f816cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:17.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.520+0000 7f09fe51b700 1 --2- 172.21.15.27:0/3851996826 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f09f80b6170 0x7f09f816cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:17.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.520+0000 7f09fe51b700 1 -- 172.21.15.27:0/3851996826 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e8009ae0 con 0x7f09f815d850 2023-10-06T01:42:17.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.520+0000 7f09fe51b700 1 --2- 172.21.15.27:0/3851996826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09f815d850 0x7f09f812fb90 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f09e800b6c0 tx=0x7f09e8018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:17.525 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.520+0000 7f09f5ffb700 1 -- 172.21.15.27:0/3851996826 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e80032f0 con 0x7f09f815d850 2023-10-06T01:42:17.525 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.520+0000 7f09f5ffb700 1 -- 172.21.15.27:0/3851996826 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09e8008d20 con 0x7f09f815d850 2023-10-06T01:42:17.525 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.520+0000 7f09fed1c700 1 -- 172.21.15.27:0/3851996826 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09f8130510 con 0x7f09f815d850 2023-10-06T01:42:17.525 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.520+0000 7f09f5ffb700 1 -- 172.21.15.27:0/3851996826 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e8026670 con 0x7f09f815d850 2023-10-06T01:42:17.525 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.520+0000 7f09fed1c700 1 -- 172.21.15.27:0/3851996826 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09f8130ad0 con 0x7f09f815d850 2023-10-06T01:42:17.525 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.521+0000 7f09fed1c700 1 -- 172.21.15.27:0/3851996826 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09f804c3c0 con 0x7f09f815d850 2023-10-06T01:42:17.525 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.522+0000 7f09f5ffb700 1 -- 172.21.15.27:0/3851996826 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f09e8003040 con 0x7f09f815d850 2023-10-06T01:42:17.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.522+0000 7f09f5ffb700 1 --2- 172.21.15.27:0/3851996826 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f09e4066f30 0x7f09e40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:17.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.522+0000 7f09f5ffb700 1 -- 172.21.15.27:0/3851996826 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f09e8086800 con 0x7f09f815d850 2023-10-06T01:42:17.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.523+0000 7f09fdd1a700 1 --2- 172.21.15.27:0/3851996826 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f09e4066f30 0x7f09e40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:17.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.523+0000 7f09fdd1a700 1 --2- 172.21.15.27:0/3851996826 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f09e4066f30 0x7f09e40693e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f09f0003a80 tx=0x7f09f0007ec0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:17.527 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.527+0000 7f09f5ffb700 1 -- 172.21.15.27:0/3851996826 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f09e80536e0 con 0x7f09f815d850 2023-10-06T01:42:17.768 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.767+0000 7f09fed1c700 1 -- 172.21.15.27:0/3851996826 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09f8069060 con 0x7f09f815d850 2023-10-06T01:42:17.769 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.768+0000 7f09f5ffb700 1 -- 172.21.15.27:0/3851996826 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4599 (secure 0 0 0) 0x7f09e8056d00 con 0x7f09f815d850 2023-10-06T01:42:17.769 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:17.770 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4870,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6836/1043044736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":1043044736},{"type":"v1","addr":"172.21.15.124:6836","nonce":1043044736}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4962,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/1207987117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1207987117},{"type":"v1","addr":"172.21.15.27:6834","nonce":1207987117}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4965,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/943261840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":943261840},{"type":"v1","addr":"172.21.15.27:6835","nonce":943261840}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T01:42:03.036784+0000","modified":"2023-10-06T01:42:17.363571+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4974},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4974":{"gid":4974,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.124:6837/325247112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6835","nonce":325247112},{"type":"v1","addr":"172.21.15.124:6837","nonce":325247112}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6,7],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-10-06T01:42:17.771 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.770+0000 7f09fed1c700 1 -- 172.21.15.27:0/3851996826 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f09e4066f30 msgr2=0x7f09e40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:17.771 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.770+0000 7f09fed1c700 1 --2- 172.21.15.27:0/3851996826 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f09e4066f30 0x7f09e40693e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f09f0003a80 tx=0x7f09f0007ec0 comp rx=0 tx=0).stop 2023-10-06T01:42:17.771 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.770+0000 7f09fed1c700 1 -- 172.21.15.27:0/3851996826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09f815d850 msgr2=0x7f09f812fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:17.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.770+0000 7f09fed1c700 1 --2- 172.21.15.27:0/3851996826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09f815d850 0x7f09f812fb90 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f09e800b6c0 tx=0x7f09e8018000 comp rx=0 tx=0).stop 2023-10-06T01:42:17.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.770+0000 7f09fed1c700 1 -- 172.21.15.27:0/3851996826 shutdown_connections 2023-10-06T01:42:17.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.770+0000 7f09fed1c700 1 --2- 172.21.15.27:0/3851996826 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f09e4066f30 0x7f09e40693e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:17.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.770+0000 7f09fed1c700 1 --2- 172.21.15.27:0/3851996826 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f09f815d850 0x7f09f812fb90 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:17.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.770+0000 7f09fed1c700 1 --2- 172.21.15.27:0/3851996826 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f09f80b6170 0x7f09f816cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:17.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.770+0000 7f09fed1c700 1 --2- 172.21.15.27:0/3851996826 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f09f815d480 0x7f09f812f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:17.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.770+0000 7f09fed1c700 1 -- 172.21.15.27:0/3851996826 >> 172.21.15.27:0/3851996826 conn(0x7f09f80ccf80 msgr2=0x7f09f8141bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:17.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.770+0000 7f09fed1c700 1 -- 172.21.15.27:0/3851996826 shutdown_connections 2023-10-06T01:42:17.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.770+0000 7f09fed1c700 1 -- 172.21.15.27:0/3851996826 wait complete. 2023-10-06T01:42:17.773 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 16 2023-10-06T01:42:17.783 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T01:42:17.902 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.901+0000 7f393bc56700 1 Processor -- start 2023-10-06T01:42:17.902 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.901+0000 7f393bc56700 1 -- start start 2023-10-06T01:42:17.904 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.902+0000 7f393bc56700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f39340cdc00 0x7f39340cbe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:17.904 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.902+0000 7f393bc56700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f39340c6dc0 0x7f39340ae5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:17.904 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.902+0000 7f393bc56700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39340ae140 con 0x7f39340d1030 2023-10-06T01:42:17.904 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.902+0000 7f393bc56700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39340cc390 con 0x7f39340cdc00 2023-10-06T01:42:17.904 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.902+0000 7f393bc56700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39340ad530 con 0x7f39340c6dc0 2023-10-06T01:42:17.905 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.902+0000 7f39391f1700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f39340d1030 0x7f39340c66b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40358/0 (socket says 172.21.15.27:40358) 2023-10-06T01:42:17.905 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.902+0000 7f39391f1700 1 -- 172.21.15.27:0/2405938510 learned_addr learned my addr 172.21.15.27:0/2405938510 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:17.905 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.902+0000 7f39399f2700 1 --2- 172.21.15.27:0/2405938510 >> v2:172.21.15.124:3300/0 conn(0x7f39340cdc00 0x7f39340cbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:17.905 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.902+0000 7f393a1f3700 1 --2- 172.21.15.27:0/2405938510 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f39340c6dc0 0x7f39340ae5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:17.905 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.902+0000 7f39389f0700 1 -- 172.21.15.27:0/2405938510 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1631100850 0 0) 0x7f39340ae140 con 0x7f39340d1030 2023-10-06T01:42:17.905 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.902+0000 7f39389f0700 1 -- 172.21.15.27:0/2405938510 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3920003570 con 0x7f39340d1030 2023-10-06T01:42:17.905 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.902+0000 7f39399f2700 1 -- 172.21.15.27:0/2405938510 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f39340c6dc0 msgr2=0x7f39340ae5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:17.905 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.902+0000 7f39399f2700 1 --2- 172.21.15.27:0/2405938510 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f39340c6dc0 0x7f39340ae5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:17.905 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.902+0000 7f39399f2700 1 -- 172.21.15.27:0/2405938510 >> v1:172.21.15.27:6789/0 conn(0x7f39340d1030 legacy=0x7f39340c66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:17.905 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.902+0000 7f39399f2700 1 -- 172.21.15.27:0/2405938510 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39340ad810 con 0x7f39340cdc00 2023-10-06T01:42:17.905 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.903+0000 7f39399f2700 1 --2- 172.21.15.27:0/2405938510 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f39340cdc00 0x7f39340cbe50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f39300098d0 tx=0x7f3930009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=506695e8bc5ea7c6 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:17.905 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.903+0000 7f39389f0700 1 -- 172.21.15.27:0/2405938510 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3930005c10 con 0x7f39340cdc00 2023-10-06T01:42:17.906 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.903+0000 7f39389f0700 1 -- 172.21.15.27:0/2405938510 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3930005d70 con 0x7f39340cdc00 2023-10-06T01:42:17.906 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.903+0000 7f39389f0700 1 -- 172.21.15.27:0/2405938510 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3930006280 con 0x7f39340cdc00 2023-10-06T01:42:17.908 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.903+0000 7f393bc56700 1 -- 172.21.15.27:0/2405938510 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f39340cdc00 msgr2=0x7f39340cbe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:17.908 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.903+0000 7f393bc56700 1 --2- 172.21.15.27:0/2405938510 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f39340cdc00 0x7f39340cbe50 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f39300098d0 tx=0x7f3930009120 comp rx=0 tx=0).stop 2023-10-06T01:42:17.908 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.903+0000 7f393bc56700 1 -- 172.21.15.27:0/2405938510 shutdown_connections 2023-10-06T01:42:17.908 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.903+0000 7f393bc56700 1 --2- 172.21.15.27:0/2405938510 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f39340c6dc0 0x7f39340ae5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:17.908 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.903+0000 7f393bc56700 1 --2- 172.21.15.27:0/2405938510 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f39340cdc00 0x7f39340cbe50 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:17.909 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.903+0000 7f393bc56700 1 -- 172.21.15.27:0/2405938510 >> 172.21.15.27:0/2405938510 conn(0x7f3934099990 msgr2=0x7f39340b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:17.909 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.903+0000 7f393bc56700 1 -- 172.21.15.27:0/2405938510 shutdown_connections 2023-10-06T01:42:17.909 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.903+0000 7f393bc56700 1 -- 172.21.15.27:0/2405938510 wait complete. 2023-10-06T01:42:17.909 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.904+0000 7f393bc56700 1 Processor -- start 2023-10-06T01:42:17.909 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.905+0000 7f393bc56700 1 -- start start 2023-10-06T01:42:17.909 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.905+0000 7f393bc56700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f39340c6dc0 0x7f39340690b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:17.909 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.905+0000 7f393bc56700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39340cdc00 0x7f39340c9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:17.909 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.905+0000 7f393bc56700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f39340d1030 0x7f393409a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:17.909 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.905+0000 7f393bc56700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393409e2d0 con 0x7f39340cdc00 2023-10-06T01:42:17.909 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.905+0000 7f393bc56700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393409d750 con 0x7f39340d1030 2023-10-06T01:42:17.909 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.905+0000 7f393bc56700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f393409da30 con 0x7f39340c6dc0 2023-10-06T01:42:17.909 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.905+0000 7f39391f1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39340cdc00 0x7f39340c9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:17.910 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.905+0000 7f39391f1700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39340cdc00 0x7f39340c9a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59830/0 (socket says 172.21.15.27:59830) 2023-10-06T01:42:17.910 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.905+0000 7f39391f1700 1 -- 172.21.15.27:0/3786383482 learned_addr learned my addr 172.21.15.27:0/3786383482 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:17.910 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.906+0000 7f393a1f3700 1 --2- 172.21.15.27:0/3786383482 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f39340d1030 0x7f393409a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:17.910 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.906+0000 7f39399f2700 1 --2- 172.21.15.27:0/3786383482 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f39340c6dc0 0x7f39340690b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:17.910 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.906+0000 7f39391f1700 1 -- 172.21.15.27:0/3786383482 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f39340c6dc0 msgr2=0x7f39340690b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:17.910 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.906+0000 7f39391f1700 1 --2- 172.21.15.27:0/3786383482 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f39340c6dc0 0x7f39340690b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:17.910 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.906+0000 7f39391f1700 1 -- 172.21.15.27:0/3786383482 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f39340d1030 msgr2=0x7f393409a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:17.910 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.906+0000 7f39391f1700 1 --2- 172.21.15.27:0/3786383482 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f39340d1030 0x7f393409a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:17.910 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.906+0000 7f39391f1700 1 -- 172.21.15.27:0/3786383482 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39300093c0 con 0x7f39340cdc00 2023-10-06T01:42:17.910 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.906+0000 7f393a1f3700 1 --2- 172.21.15.27:0/3786383482 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f39340d1030 0x7f393409a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:42:17.910 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.906+0000 7f39391f1700 1 --2- 172.21.15.27:0/3786383482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39340cdc00 0x7f39340c9a00 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f39240011e0 tx=0x7f3924007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:17.911 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.906+0000 7f3929ffb700 1 -- 172.21.15.27:0/3786383482 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3924005b30 con 0x7f39340cdc00 2023-10-06T01:42:17.911 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.906+0000 7f3929ffb700 1 -- 172.21.15.27:0/3786383482 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3924005c90 con 0x7f39340cdc00 2023-10-06T01:42:17.911 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.906+0000 7f393bc56700 1 -- 172.21.15.27:0/3786383482 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f393409dd10 con 0x7f39340cdc00 2023-10-06T01:42:17.911 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.906+0000 7f3929ffb700 1 -- 172.21.15.27:0/3786383482 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3924007320 con 0x7f39340cdc00 2023-10-06T01:42:17.911 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.906+0000 7f393bc56700 1 -- 172.21.15.27:0/3786383482 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39340695f0 con 0x7f39340cdc00 2023-10-06T01:42:17.911 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.907+0000 7f393bc56700 1 -- 172.21.15.27:0/3786383482 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3934064050 con 0x7f39340cdc00 2023-10-06T01:42:17.912 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.907+0000 7f3929ffb700 1 -- 172.21.15.27:0/3786383482 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3924007690 con 0x7f39340cdc00 2023-10-06T01:42:17.912 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.908+0000 7f3929ffb700 1 --2- 172.21.15.27:0/3786383482 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3918066f30 0x7f39180693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:17.912 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.908+0000 7f3929ffb700 1 -- 172.21.15.27:0/3786383482 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f3924083d10 con 0x7f39340cdc00 2023-10-06T01:42:17.912 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.908+0000 7f39399f2700 1 --2- 172.21.15.27:0/3786383482 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3918066f30 0x7f39180693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:17.912 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.908+0000 7f39399f2700 1 --2- 172.21.15.27:0/3786383482 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3918066f30 0x7f39180693e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f39300030c0 tx=0x7f3930016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:17.913 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:17.912+0000 7f3929ffb700 1 -- 172.21.15.27:0/3786383482 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3924050c90 con 0x7f39340cdc00 2023-10-06T01:42:18.109 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.108+0000 7f393bc56700 1 -- 172.21.15.27:0/3786383482 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f393409b030 con 0x7f39340cdc00 2023-10-06T01:42:18.110 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.109+0000 7f3929ffb700 1 -- 172.21.15.27:0/3786383482 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+18438 (secure 0 0 0) 0x7f3924017020 con 0x7f39340cdc00 2023-10-06T01:42:18.111 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:18.111 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":33,"fsid":"20a7bff0-74f8-4f2f-b6c6-899e804bc965","created":"2023-10-06T01:40:54.474215+0000","modified":"2023-10-06T01:42:16.756874+0000","last_up_change":"2023-10-06T01:41:05.242847+0000","last_in_change":"2023-10-06T01:40:56.890960+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-10-06T01:41:05.685386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-10-06T01:41:06.092995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-10-06T01:42:01.295053+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-10-06T01:42:02.005152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":7,"pool_name":"data_new","create_time":"2023-10-06T01:42:16.105156+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b902c86d-800c-40c5-b152-b545e50d02c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":90644},{"type":"v1","addr":"172.21.15.27:6801","nonce":90644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":90644},{"type":"v1","addr":"172.21.15.27:6803","nonce":90644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":90644},{"type":"v1","addr":"172.21.15.27:6807","nonce":90644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":90644},{"type":"v1","addr":"172.21.15.27:6805","nonce":90644}]},"public_addr":"172.21.15.27:6801/90644","cluster_addr":"172.21.15.27:6803/90644","heartbeat_back_addr":"172.21.15.27:6807/90644","heartbeat_front_addr":"172.21.15.27:6805/90644","state":["exists","up"]},{"osd":1,"uuid":"5b4a70e5-b06e-48cd-bfc4-7962c5d05386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":90643},{"type":"v1","addr":"172.21.15.27:6809","nonce":90643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":90643},{"type":"v1","addr":"172.21.15.27:6811","nonce":90643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":90643},{"type":"v1","addr":"172.21.15.27:6815","nonce":90643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":90643},{"type":"v1","addr":"172.21.15.27:6813","nonce":90643}]},"public_addr":"172.21.15.27:6809/90643","cluster_addr":"172.21.15.27:6811/90643","heartbeat_back_addr":"172.21.15.27:6815/90643","heartbeat_front_addr":"172.21.15.27:6813/90643","state":["exists","up"]},{"osd":2,"uuid":"22d9de46-4073-4336-9122-5227e601eca0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":90645},{"type":"v1","addr":"172.21.15.27:6817","nonce":90645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":90645},{"type":"v1","addr":"172.21.15.27:6819","nonce":90645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":90645},{"type":"v1","addr":"172.21.15.27:6823","nonce":90645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":90645},{"type":"v1","addr":"172.21.15.27:6821","nonce":90645}]},"public_addr":"172.21.15.27:6817/90645","cluster_addr":"172.21.15.27:6819/90645","heartbeat_back_addr":"172.21.15.27:6823/90645","heartbeat_front_addr":"172.21.15.27:6821/90645","state":["exists","up"]},{"osd":3,"uuid":"438bb2d3-34e2-4b7c-9734-8360c1b67650","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":90668},{"type":"v1","addr":"172.21.15.27:6825","nonce":90668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":90668},{"type":"v1","addr":"172.21.15.27:6827","nonce":90668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":90668},{"type":"v1","addr":"172.21.15.27:6831","nonce":90668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":90668},{"type":"v1","addr":"172.21.15.27:6829","nonce":90668}]},"public_addr":"172.21.15.27:6825/90668","cluster_addr":"172.21.15.27:6827/90668","heartbeat_back_addr":"172.21.15.27:6831/90668","heartbeat_front_addr":"172.21.15.27:6829/90668","state":["exists","up"]},{"osd":4,"uuid":"e1956a8e-e0f8-4e7c-acbb-40037415a354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":90863},{"type":"v1","addr":"172.21.15.124:6801","nonce":90863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":90863},{"type":"v1","addr":"172.21.15.124:6803","nonce":90863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":90863},{"type":"v1","addr":"172.21.15.124:6807","nonce":90863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":90863},{"type":"v1","addr":"172.21.15.124:6805","nonce":90863}]},"public_addr":"172.21.15.124:6801/90863","cluster_addr":"172.21.15.124:6803/90863","heartbeat_back_addr":"172.21.15.124:6807/90863","heartbeat_front_addr":"172.21.15.124:6805/90863","state":["exists","up"]},{"osd":5,"uuid":"306c59d4-846d-4b73-b1c9-051bec9e9368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":90885},{"type":"v1","addr":"172.21.15.124:6817","nonce":90885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":90885},{"type":"v1","addr":"172.21.15.124:6819","nonce":90885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":90885},{"type":"v1","addr":"172.21.15.124:6823","nonce":90885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":90885},{"type":"v1","addr":"172.21.15.124:6821","nonce":90885}]},"public_addr":"172.21.15.124:6817/90885","cluster_addr":"172.21.15.124:6819/90885","heartbeat_back_addr":"172.21.15.124:6823/90885","heartbeat_front_addr":"172.21.15.124:6821/90885","state":["exists","up"]},{"osd":6,"uuid":"15463e53-ac8d-47d1-b4f6-c0d31f29e4a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":90864},{"type":"v1","addr":"172.21.15.124:6809","nonce":90864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":90864},{"type":"v1","addr":"172.21.15.124:6811","nonce":90864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":90864},{"type":"v1","addr":"172.21.15.124:6815","nonce":90864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":90864},{"type":"v1","addr":"172.21.15.124:6813","nonce":90864}]},"public_addr":"172.21.15.124:6809/90864","cluster_addr":"172.21.15.124:6811/90864","heartbeat_back_addr":"172.21.15.124:6815/90864","heartbeat_front_addr":"172.21.15.124:6813/90864","state":["exists","up"]},{"osd":7,"uuid":"ae463a90-a9d3-498f-86fa-47a35b3fe889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":90894},{"type":"v1","addr":"172.21.15.124:6825","nonce":90894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":90894},{"type":"v1","addr":"172.21.15.124:6827","nonce":90894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":90894},{"type":"v1","addr":"172.21.15.124:6831","nonce":90894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":90894},{"type":"v1","addr":"172.21.15.124:6829","nonce":90894}]},"public_addr":"172.21.15.124:6825/90894","cluster_addr":"172.21.15.124:6827/90894","heartbeat_back_addr":"172.21.15.124:6831/90894","heartbeat_front_addr":"172.21.15.124:6829/90894","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.827704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.839951+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.032230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.050605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.987908+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.031290+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.080707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.053692+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T01:42:18.113 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.111+0000 7f393bc56700 1 -- 172.21.15.27:0/3786383482 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3918066f30 msgr2=0x7f39180693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:18.113 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.111+0000 7f393bc56700 1 --2- 172.21.15.27:0/3786383482 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3918066f30 0x7f39180693e0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f39300030c0 tx=0x7f3930016000 comp rx=0 tx=0).stop 2023-10-06T01:42:18.114 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.111+0000 7f393bc56700 1 -- 172.21.15.27:0/3786383482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39340cdc00 msgr2=0x7f39340c9a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:18.114 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.111+0000 7f393bc56700 1 --2- 172.21.15.27:0/3786383482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39340cdc00 0x7f39340c9a00 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f39240011e0 tx=0x7f3924007df0 comp rx=0 tx=0).stop 2023-10-06T01:42:18.114 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.111+0000 7f393bc56700 1 -- 172.21.15.27:0/3786383482 shutdown_connections 2023-10-06T01:42:18.114 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.111+0000 7f393bc56700 1 --2- 172.21.15.27:0/3786383482 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3918066f30 0x7f39180693e0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:18.114 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.111+0000 7f393bc56700 1 --2- 172.21.15.27:0/3786383482 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f39340c6dc0 0x7f39340690b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:18.115 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.111+0000 7f393bc56700 1 --2- 172.21.15.27:0/3786383482 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f39340d1030 0x7f393409a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:18.115 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.111+0000 7f393bc56700 1 --2- 172.21.15.27:0/3786383482 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f39340cdc00 0x7f39340c9a00 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:18.115 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.111+0000 7f393bc56700 1 -- 172.21.15.27:0/3786383482 >> 172.21.15.27:0/3786383482 conn(0x7f3934099990 msgr2=0x7f39340cfc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:18.115 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.111+0000 7f393bc56700 1 -- 172.21.15.27:0/3786383482 shutdown_connections 2023-10-06T01:42:18.115 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.111+0000 7f393bc56700 1 -- 172.21.15.27:0/3786383482 wait complete. 2023-10-06T01:42:18.124 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T01:42:18.125 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.file.layout.pool -v data_new ./parent_b/alpha) 2023-10-06T01:42:18.155 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T01:42:18.314 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.313+0000 7f7a850b8700 1 Processor -- start 2023-10-06T01:42:18.315 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.313+0000 7f7a850b8700 1 -- start start 2023-10-06T01:42:18.321 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.314+0000 7f7a850b8700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7a80142d90 0x7f7a8006a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:18.321 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.314+0000 7f7a850b8700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a80069f50 con 0x7f7a800cc130 2023-10-06T01:42:18.321 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.314+0000 7f7a850b8700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a8006a920 con 0x7f7a80143160 2023-10-06T01:42:18.321 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.314+0000 7f7a850b8700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a80069340 con 0x7f7a80142d90 2023-10-06T01:42:18.321 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.314+0000 7f7a7f59e700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7a800cc130 0x7f7a800c8b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40366/0 (socket says 172.21.15.27:40366) 2023-10-06T01:42:18.321 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.314+0000 7f7a7f59e700 1 -- 172.21.15.27:0/1213972835 learned_addr learned my addr 172.21.15.27:0/1213972835 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:18.321 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.314+0000 7f7a7ed9d700 1 --2- 172.21.15.27:0/1213972835 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7a80142d90 0x7f7a8006a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:18.322 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.314+0000 7f7a7dd9b700 1 -- 172.21.15.27:0/1213972835 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3051554975 0 0) 0x7f7a80069f50 con 0x7f7a800cc130 2023-10-06T01:42:18.322 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.315+0000 7f7a7dd9b700 1 -- 172.21.15.27:0/1213972835 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a6c003570 con 0x7f7a800cc130 2023-10-06T01:42:18.322 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.315+0000 7f7a7dd9b700 1 -- 172.21.15.27:0/1213972835 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3031314654 0 0) 0x7f7a6c003570 con 0x7f7a800cc130 2023-10-06T01:42:18.322 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.315+0000 7f7a7dd9b700 1 -- 172.21.15.27:0/1213972835 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7a80142d90 msgr2=0x7f7a8006a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:18.322 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.315+0000 7f7a7dd9b700 1 --2- 172.21.15.27:0/1213972835 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7a80142d90 0x7f7a8006a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:18.322 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.315+0000 7f7a7dd9b700 1 -- 172.21.15.27:0/1213972835 >> v1:172.21.15.124:6789/0 conn(0x7f7a80143160 legacy=0x7f7a800c43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:18.322 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.315+0000 7f7a7dd9b700 1 -- 172.21.15.27:0/1213972835 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a80069620 con 0x7f7a800cc130 2023-10-06T01:42:18.322 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.315+0000 7f7a7dd9b700 1 -- 172.21.15.27:0/1213972835 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2619078944 0 0) 0x7f7a780017f0 con 0x7f7a800cc130 2023-10-06T01:42:18.323 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.315+0000 7f7a7dd9b700 1 -- 172.21.15.27:0/1213972835 >> v1:172.21.15.27:6789/0 conn(0x7f7a800cc130 legacy=0x7f7a800c8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:18.323 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.315+0000 7f7a7dd9b700 1 --2- 172.21.15.27:0/1213972835 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7a80142d90 0x7f7a6c003a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:18.323 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.315+0000 7f7a7dd9b700 1 --2- 172.21.15.27:0/1213972835 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7a6c004010 0x7f7a6c004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:18.323 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.315+0000 7f7a7dd9b700 1 --2- 172.21.15.27:0/1213972835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a6c004d50 0x7f7a6c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:18.323 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.315+0000 7f7a7dd9b700 1 -- 172.21.15.27:0/1213972835 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a80069340 con 0x7f7a6c004d50 2023-10-06T01:42:18.323 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.315+0000 7f7a7dd9b700 1 -- 172.21.15.27:0/1213972835 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c00bbf0 con 0x7f7a6c004010 2023-10-06T01:42:18.323 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.315+0000 7f7a7dd9b700 1 -- 172.21.15.27:0/1213972835 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6c0145c0 con 0x7f7a80142d90 2023-10-06T01:42:18.323 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.315+0000 7f7a7f59e700 1 --2- 172.21.15.27:0/1213972835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a6c004d50 0x7f7a6c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:18.323 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.315+0000 7f7a7ed9d700 1 --2- 172.21.15.27:0/1213972835 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7a80142d90 0x7f7a6c003a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:18.323 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.315+0000 7f7a7e59c700 1 --2- 172.21.15.27:0/1213972835 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7a6c004010 0x7f7a6c004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:18.323 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.315+0000 7f7a7ed9d700 1 -- 172.21.15.27:0/1213972835 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7a6c004010 msgr2=0x7f7a6c004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:18.324 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.315+0000 7f7a7ed9d700 1 --2- 172.21.15.27:0/1213972835 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7a6c004010 0x7f7a6c004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:18.324 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.315+0000 7f7a7ed9d700 1 -- 172.21.15.27:0/1213972835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a6c004d50 msgr2=0x7f7a6c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:18.324 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.315+0000 7f7a7ed9d700 1 --2- 172.21.15.27:0/1213972835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a6c004d50 0x7f7a6c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:18.324 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.315+0000 7f7a7ed9d700 1 -- 172.21.15.27:0/1213972835 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a6c003570 con 0x7f7a80142d90 2023-10-06T01:42:18.324 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.315+0000 7f7a7f59e700 1 --2- 172.21.15.27:0/1213972835 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a6c004d50 0x7f7a6c0051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:42:18.324 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.316+0000 7f7a7ed9d700 1 --2- 172.21.15.27:0/1213972835 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7a80142d90 0x7f7a6c003a90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7a70006f00 tx=0x7f7a700091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9785a2ec2f0284d5 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:18.324 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.316+0000 7f7a7dd9b700 1 -- 172.21.15.27:0/1213972835 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7000e710 con 0x7f7a80142d90 2023-10-06T01:42:18.324 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.316+0000 7f7a7dd9b700 1 -- 172.21.15.27:0/1213972835 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a7000e870 con 0x7f7a80142d90 2023-10-06T01:42:18.324 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.316+0000 7f7a7dd9b700 1 -- 172.21.15.27:0/1213972835 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7000ec30 con 0x7f7a80142d90 2023-10-06T01:42:18.324 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.316+0000 7f7a850b8700 1 -- 172.21.15.27:0/1213972835 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7a80142d90 msgr2=0x7f7a6c003a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:18.325 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.316+0000 7f7a850b8700 1 --2- 172.21.15.27:0/1213972835 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7a80142d90 0x7f7a6c003a90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7a70006f00 tx=0x7f7a700091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:18.325 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.316+0000 7f7a7f59e700 1 -- 172.21.15.27:0/1213972835 reap_dead start 2023-10-06T01:42:18.325 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.316+0000 7f7a850b8700 1 -- 172.21.15.27:0/1213972835 shutdown_connections 2023-10-06T01:42:18.325 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.316+0000 7f7a850b8700 1 -- 172.21.15.27:0/1213972835 >> 172.21.15.27:0/1213972835 conn(0x7f7a800ce4d0 msgr2=0x7f7a800ce8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:18.325 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.316+0000 7f7a850b8700 1 -- 172.21.15.27:0/1213972835 shutdown_connections 2023-10-06T01:42:18.325 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.316+0000 7f7a850b8700 1 -- 172.21.15.27:0/1213972835 wait complete. 2023-10-06T01:42:18.325 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.317+0000 7f7a850b8700 1 Processor -- start 2023-10-06T01:42:18.325 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.318+0000 7f7a850b8700 1 -- start start 2023-10-06T01:42:18.325 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.318+0000 7f7a850b8700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7a80142d90 0x7f7a801404a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:18.325 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.318+0000 7f7a850b8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a8013fd40 0x7f7a8013de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:18.325 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.318+0000 7f7a850b8700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7a80165380 0x7f7a801657f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:18.326 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.318+0000 7f7a850b8700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a80158c50 con 0x7f7a8013fd40 2023-10-06T01:42:18.326 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.318+0000 7f7a850b8700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a80140ba0 con 0x7f7a80142d90 2023-10-06T01:42:18.326 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.318+0000 7f7a850b8700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a80140e80 con 0x7f7a80165380 2023-10-06T01:42:18.326 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.318+0000 7f7a7e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a8013fd40 0x7f7a8013de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:18.326 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.318+0000 7f7a7e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a8013fd40 0x7f7a8013de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59848/0 (socket says 172.21.15.27:59848) 2023-10-06T01:42:18.326 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.318+0000 7f7a7e59c700 1 -- 172.21.15.27:0/3419998666 learned_addr learned my addr 172.21.15.27:0/3419998666 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:18.326 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.318+0000 7f7a7f59e700 1 --2- 172.21.15.27:0/3419998666 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7a80165380 0x7f7a801657f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:18.326 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.318+0000 7f7a7ed9d700 1 --2- 172.21.15.27:0/3419998666 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7a80142d90 0x7f7a801404a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:18.326 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.319+0000 7f7a7e59c700 1 -- 172.21.15.27:0/3419998666 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7a80165380 msgr2=0x7f7a801657f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:18.327 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.319+0000 7f7a7e59c700 1 --2- 172.21.15.27:0/3419998666 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7a80165380 0x7f7a801657f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:18.327 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.319+0000 7f7a7e59c700 1 -- 172.21.15.27:0/3419998666 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7a80142d90 msgr2=0x7f7a801404a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:18.327 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.319+0000 7f7a7e59c700 1 --2- 172.21.15.27:0/3419998666 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7a80142d90 0x7f7a801404a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:18.327 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.319+0000 7f7a7e59c700 1 -- 172.21.15.27:0/3419998666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a80158600 con 0x7f7a8013fd40 2023-10-06T01:42:18.327 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.319+0000 7f7a7e59c700 1 --2- 172.21.15.27:0/3419998666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a8013fd40 0x7f7a8013de60 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f7a68002920 tx=0x7f7a68007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:18.327 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.319+0000 7f7a76ffd700 1 -- 172.21.15.27:0/3419998666 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6800f8d0 con 0x7f7a8013fd40 2023-10-06T01:42:18.328 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.319+0000 7f7a76ffd700 1 -- 172.21.15.27:0/3419998666 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a6800fa30 con 0x7f7a8013fd40 2023-10-06T01:42:18.328 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.319+0000 7f7a76ffd700 1 -- 172.21.15.27:0/3419998666 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a6800fdf0 con 0x7f7a8013fd40 2023-10-06T01:42:18.328 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.319+0000 7f7a850b8700 1 -- 172.21.15.27:0/3419998666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a801588e0 con 0x7f7a8013fd40 2023-10-06T01:42:18.328 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.319+0000 7f7a850b8700 1 -- 172.21.15.27:0/3419998666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a8013e3a0 con 0x7f7a8013fd40 2023-10-06T01:42:18.329 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.320+0000 7f7a850b8700 1 -- 172.21.15.27:0/3419998666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a80064050 con 0x7f7a8013fd40 2023-10-06T01:42:18.330 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.320+0000 7f7a76ffd700 1 -- 172.21.15.27:0/3419998666 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7a68013600 con 0x7f7a8013fd40 2023-10-06T01:42:18.330 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.321+0000 7f7a76ffd700 1 --2- 172.21.15.27:0/3419998666 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7a60066f30 0x7f7a600693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:18.330 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.321+0000 7f7a76ffd700 1 -- 172.21.15.27:0/3419998666 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f7a68084dd0 con 0x7f7a8013fd40 2023-10-06T01:42:18.330 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.321+0000 7f7a7ed9d700 1 --2- 172.21.15.27:0/3419998666 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7a60066f30 0x7f7a600693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:18.330 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.321+0000 7f7a7ed9d700 1 --2- 172.21.15.27:0/3419998666 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7a60066f30 0x7f7a600693e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7a700091e0 tx=0x7f7a7001a000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:18.330 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.325+0000 7f7a76ffd700 1 -- 172.21.15.27:0/3419998666 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7a68051cb0 con 0x7f7a8013fd40 2023-10-06T01:42:18.568 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.567+0000 7f7a850b8700 1 -- 172.21.15.27:0/3419998666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a80069900 con 0x7f7a8013fd40 2023-10-06T01:42:18.569 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.567+0000 7f7a76ffd700 1 -- 172.21.15.27:0/3419998666 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4599 (secure 0 0 0) 0x7f7a680552d0 con 0x7f7a8013fd40 2023-10-06T01:42:18.569 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:18.569 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4870,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6836/1043044736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":1043044736},{"type":"v1","addr":"172.21.15.124:6836","nonce":1043044736}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4962,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/1207987117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1207987117},{"type":"v1","addr":"172.21.15.27:6834","nonce":1207987117}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4965,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/943261840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":943261840},{"type":"v1","addr":"172.21.15.27:6835","nonce":943261840}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T01:42:03.036784+0000","modified":"2023-10-06T01:42:17.363571+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4974},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4974":{"gid":4974,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.124:6837/325247112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6835","nonce":325247112},{"type":"v1","addr":"172.21.15.124:6837","nonce":325247112}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6,7],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-10-06T01:42:18.571 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.569+0000 7f7a850b8700 1 -- 172.21.15.27:0/3419998666 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7a60066f30 msgr2=0x7f7a600693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:18.571 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.569+0000 7f7a850b8700 1 --2- 172.21.15.27:0/3419998666 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7a60066f30 0x7f7a600693e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7a700091e0 tx=0x7f7a7001a000 comp rx=0 tx=0).stop 2023-10-06T01:42:18.571 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.569+0000 7f7a850b8700 1 -- 172.21.15.27:0/3419998666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a8013fd40 msgr2=0x7f7a8013de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:18.571 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.569+0000 7f7a850b8700 1 --2- 172.21.15.27:0/3419998666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a8013fd40 0x7f7a8013de60 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f7a68002920 tx=0x7f7a68007d30 comp rx=0 tx=0).stop 2023-10-06T01:42:18.571 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.569+0000 7f7a850b8700 1 -- 172.21.15.27:0/3419998666 shutdown_connections 2023-10-06T01:42:18.571 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.569+0000 7f7a850b8700 1 --2- 172.21.15.27:0/3419998666 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7a60066f30 0x7f7a600693e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:18.571 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.569+0000 7f7a850b8700 1 --2- 172.21.15.27:0/3419998666 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7a80165380 0x7f7a801657f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:18.572 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.569+0000 7f7a850b8700 1 --2- 172.21.15.27:0/3419998666 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7a80142d90 0x7f7a801404a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:18.572 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.569+0000 7f7a850b8700 1 --2- 172.21.15.27:0/3419998666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7a8013fd40 0x7f7a8013de60 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:18.572 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.569+0000 7f7a850b8700 1 -- 172.21.15.27:0/3419998666 >> 172.21.15.27:0/3419998666 conn(0x7f7a800ce4d0 msgr2=0x7f7a8006c9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:18.572 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.570+0000 7f7a850b8700 1 -- 172.21.15.27:0/3419998666 shutdown_connections 2023-10-06T01:42:18.572 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.570+0000 7f7a850b8700 1 -- 172.21.15.27:0/3419998666 wait complete. 2023-10-06T01:42:18.572 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 16 2023-10-06T01:42:18.583 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json flush journal 2023-10-06T01:42:18.645 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T01:42:18.647 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T01:42:18.740 INFO:teuthology.orchestra.run.smithi124.stdout:{"message":"","return_code":0} 2023-10-06T01:42:18.740 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2023-10-06T01:42:18.740 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001fb.00000000 parent 2023-10-06T01:42:18.779 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.778+0000 7f5886f80000 1 Processor -- start 2023-10-06T01:42:18.779 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.778+0000 7f5886f80000 1 -- start start 2023-10-06T01:42:18.781 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.779+0000 7f5886f80000 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x56511f2baa70 0x56511f2e4580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:18.781 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.779+0000 7f5886f80000 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56511f371a00 0x56511f2b60a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:18.782 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.779+0000 7f5886f80000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56511f371dd0 0x56511f2b65e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:18.782 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.779+0000 7f5886f80000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x56511f25e320 con 0x56511f371dd0 2023-10-06T01:42:18.782 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.779+0000 7f5886f80000 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x56511f202980 con 0x56511f371a00 2023-10-06T01:42:18.782 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.779+0000 7f5886f80000 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x56511f202620 con 0x56511f2baa70 2023-10-06T01:42:18.782 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.779+0000 7f58809bc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56511f371dd0 0x56511f2b65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:18.782 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.779+0000 7f58809bc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56511f371dd0 0x56511f2b65e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59858/0 (socket says 172.21.15.27:59858) 2023-10-06T01:42:18.782 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.779+0000 7f58809bc700 1 -- 172.21.15.27:0/4220467051 learned_addr learned my addr 172.21.15.27:0/4220467051 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:18.782 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.779+0000 7f587f9ba700 1 --2- 172.21.15.27:0/4220467051 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56511f371a00 0x56511f2b60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:18.782 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.779+0000 7f58801bb700 1 --2- 172.21.15.27:0/4220467051 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x56511f2baa70 0x56511f2e4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:18.782 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.779+0000 7f58809bc700 1 -- 172.21.15.27:0/4220467051 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x56511f2baa70 msgr2=0x56511f2e4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:18.782 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.779+0000 7f58809bc700 1 --2- 172.21.15.27:0/4220467051 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x56511f2baa70 0x56511f2e4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:18.783 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.779+0000 7f58809bc700 1 -- 172.21.15.27:0/4220467051 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56511f371a00 msgr2=0x56511f2b60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:18.783 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.779+0000 7f58809bc700 1 --2- 172.21.15.27:0/4220467051 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56511f371a00 0x56511f2b60a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:18.783 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.779+0000 7f58809bc700 1 -- 172.21.15.27:0/4220467051 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5874005cf0 con 0x56511f371dd0 2023-10-06T01:42:18.783 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.779+0000 7f58801bb700 1 --2- 172.21.15.27:0/4220467051 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x56511f2baa70 0x56511f2e4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:42:18.783 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.779+0000 7f587f9ba700 1 --2- 172.21.15.27:0/4220467051 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56511f371a00 0x56511f2b60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:42:18.783 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.779+0000 7f58809bc700 1 --2- 172.21.15.27:0/4220467051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56511f371dd0 0x56511f2b65e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f5878006720 tx=0x7f5878001d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c46f53b6c007f03c server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:18.783 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.779+0000 7f587e9b8700 1 -- 172.21.15.27:0/4220467051 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58780062f0 con 0x56511f371dd0 2023-10-06T01:42:18.783 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.779+0000 7f587e9b8700 1 -- 172.21.15.27:0/4220467051 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5878003450 con 0x56511f371dd0 2023-10-06T01:42:18.783 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.779+0000 7f587e9b8700 1 -- 172.21.15.27:0/4220467051 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5878003830 con 0x56511f371dd0 2023-10-06T01:42:18.783 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.780+0000 7f5886f80000 1 -- 172.21.15.27:0/4220467051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56511f371dd0 msgr2=0x56511f2b65e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:18.784 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.780+0000 7f5886f80000 1 --2- 172.21.15.27:0/4220467051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56511f371dd0 0x56511f2b65e0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f5878006720 tx=0x7f5878001d00 comp rx=0 tx=0).stop 2023-10-06T01:42:18.784 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.780+0000 7f5886f80000 1 -- 172.21.15.27:0/4220467051 shutdown_connections 2023-10-06T01:42:18.784 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.780+0000 7f5886f80000 1 --2- 172.21.15.27:0/4220467051 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x56511f2baa70 0x56511f2e4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:18.784 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.780+0000 7f5886f80000 1 --2- 172.21.15.27:0/4220467051 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56511f371dd0 0x56511f2b65e0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:18.784 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.780+0000 7f5886f80000 1 --2- 172.21.15.27:0/4220467051 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56511f371a00 0x56511f2b60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:18.784 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.780+0000 7f5886f80000 1 -- 172.21.15.27:0/4220467051 >> 172.21.15.27:0/4220467051 conn(0x56511f2e77b0 msgr2=0x56511f2b8b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:18.784 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.780+0000 7f5886f80000 1 -- 172.21.15.27:0/4220467051 shutdown_connections 2023-10-06T01:42:18.784 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.780+0000 7f5886f80000 1 -- 172.21.15.27:0/4220467051 wait complete. 2023-10-06T01:42:18.784 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.780+0000 7f5886f80000 1 Processor -- start 2023-10-06T01:42:18.784 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.780+0000 7f5886f80000 1 -- start start 2023-10-06T01:42:18.788 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.781+0000 7f5886f80000 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x56511f2baa70 0x56511f38ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:18.788 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.781+0000 7f5886f80000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56511f371a00 0x56511f2dbb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:18.788 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.781+0000 7f5886f80000 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56511f371dd0 0x56511f2dd760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:18.788 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.781+0000 7f5886f80000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x56511f1f9400 con 0x56511f371a00 2023-10-06T01:42:18.788 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.781+0000 7f5886f80000 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x56511f2dac50 con 0x56511f371dd0 2023-10-06T01:42:18.788 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.781+0000 7f5886f80000 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x56511f2daef0 con 0x56511f2baa70 2023-10-06T01:42:18.788 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.781+0000 7f587f9ba700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56511f371a00 0x56511f2dbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:18.788 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.781+0000 7f587f9ba700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56511f371a00 0x56511f2dbb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59868/0 (socket says 172.21.15.27:59868) 2023-10-06T01:42:18.788 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.781+0000 7f587f9ba700 1 -- 172.21.15.27:0/1706587167 learned_addr learned my addr 172.21.15.27:0/1706587167 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:18.788 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.781+0000 7f58801bb700 1 --2- 172.21.15.27:0/1706587167 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x56511f2baa70 0x56511f38ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:18.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.781+0000 7f58809bc700 1 --2- 172.21.15.27:0/1706587167 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56511f371dd0 0x56511f2dd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:18.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.782+0000 7f587f9ba700 1 -- 172.21.15.27:0/1706587167 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x56511f2baa70 msgr2=0x56511f38ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:18.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.782+0000 7f587f9ba700 1 --2- 172.21.15.27:0/1706587167 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x56511f2baa70 0x56511f38ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:18.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.782+0000 7f587f9ba700 1 -- 172.21.15.27:0/1706587167 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56511f371dd0 msgr2=0x56511f2dd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:18.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.782+0000 7f587f9ba700 1 --2- 172.21.15.27:0/1706587167 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x56511f371dd0 0x56511f2dd760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:18.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.782+0000 7f587f9ba700 1 -- 172.21.15.27:0/1706587167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5878001fe0 con 0x56511f371a00 2023-10-06T01:42:18.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.782+0000 7f587f9ba700 1 --2- 172.21.15.27:0/1706587167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56511f371a00 0x56511f2dbb70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f5874002000 tx=0x7f5874013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:18.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.782+0000 7f587d1b5700 1 -- 172.21.15.27:0/1706587167 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58740053b0 con 0x56511f371a00 2023-10-06T01:42:18.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.782+0000 7f587d1b5700 1 -- 172.21.15.27:0/1706587167 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5874007710 con 0x56511f371a00 2023-10-06T01:42:18.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.782+0000 7f5886f80000 1 -- 172.21.15.27:0/1706587167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x56511f204ae0 con 0x56511f371a00 2023-10-06T01:42:18.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.782+0000 7f587d1b5700 1 -- 172.21.15.27:0/1706587167 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5874007ab0 con 0x56511f371a00 2023-10-06T01:42:18.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.782+0000 7f5886f80000 1 -- 172.21.15.27:0/1706587167 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56511f204540 con 0x56511f371a00 2023-10-06T01:42:18.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.783+0000 7f587d1b5700 1 -- 172.21.15.27:0/1706587167 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f587401e440 con 0x56511f371a00 2023-10-06T01:42:18.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.784+0000 7f587d1b5700 1 --2- 172.21.15.27:0/1706587167 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f586c065890 0x7f586c067d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:18.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.784+0000 7f587d1b5700 1 -- 172.21.15.27:0/1706587167 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f5874084380 con 0x56511f371a00 2023-10-06T01:42:18.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.784+0000 7f58801bb700 1 --2- 172.21.15.27:0/1706587167 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f586c065890 0x7f586c067d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:18.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.784+0000 7f5886f80000 1 --2- 172.21.15.27:0/1706587167 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x56511f38db10 0x56511f38dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:18.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.784+0000 7f5886f80000 1 -- 172.21.15.27:0/1706587167 --> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] -- osd_op(unknown.0.0:1 6.13 6:ca8fdbba:::100000001fb.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e34) v8 -- 0x56511f3a2c30 con 0x56511f38db10 2023-10-06T01:42:18.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.784+0000 7f58809bc700 1 --2- 172.21.15.27:0/1706587167 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x56511f38db10 0x56511f38dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:18.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.784+0000 7f58801bb700 1 --2- 172.21.15.27:0/1706587167 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f586c065890 0x7f586c067d40 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5870000c00 tx=0x7f5870002880 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:18.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.784+0000 7f58809bc700 1 --2- 172.21.15.27:0/1706587167 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x56511f38db10 0x56511f38dee0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:18.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.785+0000 7f58809bc700 1 -- 172.21.15.27:0/1706587167 <== osd.3 v2:172.21.15.27:6824/90668 1 ==== osd_op_reply(1 100000001fb.00000000 [getxattr (103) out=103b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+103 (crc 0 0 0) 0x7f5878003730 con 0x56511f38db10 2023-10-06T01:42:18.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.785+0000 7f5886f80000 1 -- 172.21.15.27:0/1706587167 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x56511f38db10 msgr2=0x56511f38dee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:18.790 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.785+0000 7f5886f80000 1 --2- 172.21.15.27:0/1706587167 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x56511f38db10 0x56511f38dee0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:18.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.785+0000 7f5886f80000 1 -- 172.21.15.27:0/1706587167 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f586c065890 msgr2=0x7f586c067d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:18.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.785+0000 7f5886f80000 1 --2- 172.21.15.27:0/1706587167 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f586c065890 0x7f586c067d40 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f5870000c00 tx=0x7f5870002880 comp rx=0 tx=0).stop 2023-10-06T01:42:18.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.785+0000 7f5886f80000 1 -- 172.21.15.27:0/1706587167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56511f371a00 msgr2=0x56511f2dbb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:18.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.785+0000 7f5886f80000 1 --2- 172.21.15.27:0/1706587167 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x56511f371a00 0x56511f2dbb70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f5874002000 tx=0x7f5874013000 comp rx=0 tx=0).stop 2023-10-06T01:42:18.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.785+0000 7f58809bc700 1 -- 172.21.15.27:0/1706587167 reap_dead start 2023-10-06T01:42:18.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.785+0000 7f5886f80000 1 -- 172.21.15.27:0/1706587167 shutdown_connections 2023-10-06T01:42:18.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.785+0000 7f5886f80000 1 -- 172.21.15.27:0/1706587167 >> 172.21.15.27:0/1706587167 conn(0x56511f2e77b0 msgr2=0x56511f2e45f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:18.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.785+0000 7f5886f80000 1 -- 172.21.15.27:0/1706587167 shutdown_connections 2023-10-06T01:42:18.791 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.785+0000 7f5886f80000 1 -- 172.21.15.27:0/1706587167 wait complete. 2023-10-06T01:42:18.792 INFO:teuthology.orchestra.run.smithi027.stdout:a��alphaparent_b 2023-10-06T01:42:18.792 DEBUG:teuthology.orchestra.run.smithi027:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2023-10-06T01:42:18.884 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2023-10-06T01:42:18.885 INFO:teuthology.orchestra.run.smithi027.stdout: "ino": 1099511628283, 2023-10-06T01:42:18.885 INFO:teuthology.orchestra.run.smithi027.stdout: "ancestors": [ 2023-10-06T01:42:18.885 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:42:18.885 INFO:teuthology.orchestra.run.smithi027.stdout: "dirino": 1099511628284, 2023-10-06T01:42:18.885 INFO:teuthology.orchestra.run.smithi027.stdout: "dname": "alpha", 2023-10-06T01:42:18.885 INFO:teuthology.orchestra.run.smithi027.stdout: "version": 7 2023-10-06T01:42:18.885 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2023-10-06T01:42:18.885 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:42:18.885 INFO:teuthology.orchestra.run.smithi027.stdout: "dirino": 1, 2023-10-06T01:42:18.885 INFO:teuthology.orchestra.run.smithi027.stdout: "dname": "parent_b", 2023-10-06T01:42:18.886 INFO:teuthology.orchestra.run.smithi027.stdout: "version": 30 2023-10-06T01:42:18.886 INFO:teuthology.orchestra.run.smithi027.stdout: } 2023-10-06T01:42:18.886 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2023-10-06T01:42:18.886 INFO:teuthology.orchestra.run.smithi027.stdout: "pool": 7, 2023-10-06T01:42:18.886 INFO:teuthology.orchestra.run.smithi027.stdout: "old_pools": [ 2023-10-06T01:42:18.886 INFO:teuthology.orchestra.run.smithi027.stdout: 6 2023-10-06T01:42:18.886 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2023-10-06T01:42:18.886 INFO:teuthology.orchestra.run.smithi027.stdout:} 2023-10-06T01:42:18.886 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:18.887 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool data_new getxattr 100000001fb.00000000 parent 2023-10-06T01:42:18.966 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.965+0000 7fd7bffc7000 1 Processor -- start 2023-10-06T01:42:18.966 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.965+0000 7fd7bffc7000 1 -- start start 2023-10-06T01:42:18.970 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.965+0000 7fd7bffc7000 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55ba52236a70 0x55ba522f0450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:18.970 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.965+0000 7fd7bffc7000 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55ba522f1b60 0x55ba52314db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:18.970 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.965+0000 7fd7bffc7000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ba522f1f30 0x55ba523152f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:18.971 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.965+0000 7fd7bffc7000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba5217e980 con 0x55ba522f1f30 2023-10-06T01:42:18.971 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.965+0000 7fd7bffc7000 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba5217e620 con 0x55ba522f1b60 2023-10-06T01:42:18.971 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.965+0000 7fd7bffc7000 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ba52175400 con 0x55ba52236a70 2023-10-06T01:42:18.971 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.965+0000 7fd7b9a03700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ba522f1f30 0x55ba523152f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:18.971 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.965+0000 7fd7b9a03700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ba522f1f30 0x55ba523152f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59872/0 (socket says 172.21.15.27:59872) 2023-10-06T01:42:18.971 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.965+0000 7fd7b9a03700 1 -- 172.21.15.27:0/1554439161 learned_addr learned my addr 172.21.15.27:0/1554439161 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:18.971 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.966+0000 7fd7b9202700 1 --2- 172.21.15.27:0/1554439161 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55ba52236a70 0x55ba522f0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:18.972 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.966+0000 7fd7b8a01700 1 --2- 172.21.15.27:0/1554439161 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55ba522f1b60 0x55ba52314db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:18.972 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.966+0000 7fd7b9a03700 1 -- 172.21.15.27:0/1554439161 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55ba52236a70 msgr2=0x55ba522f0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:18.972 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.966+0000 7fd7b9a03700 1 --2- 172.21.15.27:0/1554439161 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55ba52236a70 0x55ba522f0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:18.972 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.966+0000 7fd7b9a03700 1 -- 172.21.15.27:0/1554439161 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55ba522f1b60 msgr2=0x55ba52314db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:18.972 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.966+0000 7fd7b9a03700 1 --2- 172.21.15.27:0/1554439161 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55ba522f1b60 0x55ba52314db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:18.972 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.966+0000 7fd7b9a03700 1 -- 172.21.15.27:0/1554439161 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7b0005c90 con 0x55ba522f1f30 2023-10-06T01:42:18.972 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.966+0000 7fd7b9202700 1 --2- 172.21.15.27:0/1554439161 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55ba52236a70 0x55ba522f0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:42:18.972 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.966+0000 7fd7b9a03700 1 --2- 172.21.15.27:0/1554439161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ba522f1f30 0x55ba523152f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac006f10 tx=0x7fd7ac0069f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f4165d9a0067124 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:18.972 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.966+0000 7fd7ab7fe700 1 -- 172.21.15.27:0/1554439161 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac0020d0 con 0x55ba522f1f30 2023-10-06T01:42:18.973 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.966+0000 7fd7ab7fe700 1 -- 172.21.15.27:0/1554439161 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7ac006320 con 0x55ba522f1f30 2023-10-06T01:42:18.973 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.966+0000 7fd7ab7fe700 1 -- 172.21.15.27:0/1554439161 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac004730 con 0x55ba522f1f30 2023-10-06T01:42:18.973 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.966+0000 7fd7bffc7000 1 -- 172.21.15.27:0/1554439161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ba522f1f30 msgr2=0x55ba523152f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:18.973 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.966+0000 7fd7bffc7000 1 --2- 172.21.15.27:0/1554439161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ba522f1f30 0x55ba523152f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac006f10 tx=0x7fd7ac0069f0 comp rx=0 tx=0).stop 2023-10-06T01:42:18.973 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.966+0000 7fd7bffc7000 1 -- 172.21.15.27:0/1554439161 shutdown_connections 2023-10-06T01:42:18.973 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.966+0000 7fd7bffc7000 1 --2- 172.21.15.27:0/1554439161 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55ba52236a70 0x55ba522f0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:18.973 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.966+0000 7fd7bffc7000 1 --2- 172.21.15.27:0/1554439161 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ba522f1f30 0x55ba523152f0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:18.973 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.966+0000 7fd7bffc7000 1 --2- 172.21.15.27:0/1554439161 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55ba522f1b60 0x55ba52314db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:18.973 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.967+0000 7fd7bffc7000 1 -- 172.21.15.27:0/1554439161 >> 172.21.15.27:0/1554439161 conn(0x55ba52235120 msgr2=0x55ba522ecc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:18.973 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.967+0000 7fd7bffc7000 1 -- 172.21.15.27:0/1554439161 shutdown_connections 2023-10-06T01:42:18.973 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.967+0000 7fd7bffc7000 1 -- 172.21.15.27:0/1554439161 wait complete. 2023-10-06T01:42:18.973 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.967+0000 7fd7bffc7000 1 Processor -- start 2023-10-06T01:42:18.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.967+0000 7fd7bffc7000 1 -- start start 2023-10-06T01:42:18.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.968+0000 7fd7bffc7000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ba52236a70 0x55ba52258730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:18.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.968+0000 7fd7bffc7000 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55ba522f1b60 0x55ba52307da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:18.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.968+0000 7fd7bffc7000 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55ba522f1f30 0x55ba522ed9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:18.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.968+0000 7fd7bffc7000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba522592e0 con 0x55ba52236a70 2023-10-06T01:42:18.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.968+0000 7fd7bffc7000 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba5231c200 con 0x55ba522f1b60 2023-10-06T01:42:18.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.968+0000 7fd7bffc7000 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ba5231c4a0 con 0x55ba522f1f30 2023-10-06T01:42:18.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.968+0000 7fd7b9202700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ba52236a70 0x55ba52258730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:18.975 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.968+0000 7fd7b9202700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ba52236a70 0x55ba52258730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59878/0 (socket says 172.21.15.27:59878) 2023-10-06T01:42:18.975 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.968+0000 7fd7b9202700 1 -- 172.21.15.27:0/1698111685 learned_addr learned my addr 172.21.15.27:0/1698111685 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:18.975 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.968+0000 7fd7b9a03700 1 --2- 172.21.15.27:0/1698111685 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55ba522f1f30 0x55ba522ed9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:18.975 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.968+0000 7fd7b8a01700 1 --2- 172.21.15.27:0/1698111685 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55ba522f1b60 0x55ba52307da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:18.975 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.968+0000 7fd7b9202700 1 -- 172.21.15.27:0/1698111685 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55ba522f1f30 msgr2=0x55ba522ed9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:18.975 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.968+0000 7fd7b9202700 1 --2- 172.21.15.27:0/1698111685 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55ba522f1f30 0x55ba522ed9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:18.975 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.968+0000 7fd7b9202700 1 -- 172.21.15.27:0/1698111685 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55ba522f1b60 msgr2=0x55ba52307da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:18.975 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.968+0000 7fd7b9202700 1 --2- 172.21.15.27:0/1698111685 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55ba522f1b60 0x55ba52307da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:18.976 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.968+0000 7fd7b9202700 1 -- 172.21.15.27:0/1698111685 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a4001e10 con 0x55ba52236a70 2023-10-06T01:42:18.976 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.968+0000 7fd7b9202700 1 --2- 172.21.15.27:0/1698111685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ba52236a70 0x55ba52258730 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fd7b4008b50 tx=0x7fd7b4005a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:18.976 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.969+0000 7fd7a9ffb700 1 -- 172.21.15.27:0/1698111685 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b4018070 con 0x55ba52236a70 2023-10-06T01:42:18.976 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.969+0000 7fd7a9ffb700 1 -- 172.21.15.27:0/1698111685 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b4010040 con 0x55ba52236a70 2023-10-06T01:42:18.976 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.969+0000 7fd7bffc7000 1 -- 172.21.15.27:0/1698111685 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55ba5218d3d0 con 0x55ba52236a70 2023-10-06T01:42:18.976 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.969+0000 7fd7a9ffb700 1 -- 172.21.15.27:0/1698111685 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b4006cd0 con 0x55ba52236a70 2023-10-06T01:42:18.976 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.969+0000 7fd7bffc7000 1 -- 172.21.15.27:0/1698111685 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ba52180ae0 con 0x55ba52236a70 2023-10-06T01:42:18.978 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.970+0000 7fd7a9ffb700 1 -- 172.21.15.27:0/1698111685 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd7b4014600 con 0x55ba52236a70 2023-10-06T01:42:18.978 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.970+0000 7fd7a9ffb700 1 --2- 172.21.15.27:0/1698111685 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd79c065890 0x7fd79c067d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:18.979 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.970+0000 7fd7a9ffb700 1 -- 172.21.15.27:0/1698111685 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6453+0+0 (secure 0 0 0) 0x7fd7b4083350 con 0x55ba52236a70 2023-10-06T01:42:18.979 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.970+0000 7fd7b8a01700 1 --2- 172.21.15.27:0/1698111685 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd79c065890 0x7fd79c067d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:18.979 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.970+0000 7fd7bffc7000 1 --2- 172.21.15.27:0/1698111685 >> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] conn(0x55ba523082e0 0x55ba5225ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:18.979 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.971+0000 7fd7bffc7000 1 -- 172.21.15.27:0/1698111685 --> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] -- osd_op(unknown.0.0:1 7.13 7:ca8fdbba:::100000001fb.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e34) v8 -- 0x55ba523198a0 con 0x55ba523082e0 2023-10-06T01:42:18.979 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.971+0000 7fd7b9a03700 1 --2- 172.21.15.27:0/1698111685 >> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] conn(0x55ba523082e0 0x55ba5225ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:18.979 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.971+0000 7fd7b8a01700 1 --2- 172.21.15.27:0/1698111685 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd79c065890 0x7fd79c067d40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0005a30 tx=0x7fd7b0014000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:18.979 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.971+0000 7fd7b9a03700 1 --2- 172.21.15.27:0/1698111685 >> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] conn(0x55ba523082e0 0x55ba5225ac10 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:18.979 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.971+0000 7fd7b9a03700 1 -- 172.21.15.27:0/1698111685 <== osd.0 v2:172.21.15.27:6800/90644 1 ==== osd_op_reply(1 100000001fb.00000000 [getxattr (103) out=103b] v0'0 uv1 ondisk = 0) v8 ==== 164+0+103 (crc 0 0 0) 0x7fd7ac0044f0 con 0x55ba523082e0 2023-10-06T01:42:18.979 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.971+0000 7fd7bffc7000 1 -- 172.21.15.27:0/1698111685 >> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] conn(0x55ba523082e0 msgr2=0x55ba5225ac10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:18.979 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.971+0000 7fd7bffc7000 1 --2- 172.21.15.27:0/1698111685 >> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] conn(0x55ba523082e0 0x55ba5225ac10 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:18.979 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.971+0000 7fd7bffc7000 1 -- 172.21.15.27:0/1698111685 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd79c065890 msgr2=0x7fd79c067d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:18.980 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.971+0000 7fd7bffc7000 1 --2- 172.21.15.27:0/1698111685 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd79c065890 0x7fd79c067d40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0005a30 tx=0x7fd7b0014000 comp rx=0 tx=0).stop 2023-10-06T01:42:18.980 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.971+0000 7fd7bffc7000 1 -- 172.21.15.27:0/1698111685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ba52236a70 msgr2=0x55ba52258730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:18.980 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.971+0000 7fd7bffc7000 1 --2- 172.21.15.27:0/1698111685 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55ba52236a70 0x55ba52258730 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fd7b4008b50 tx=0x7fd7b4005a50 comp rx=0 tx=0).stop 2023-10-06T01:42:18.980 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.971+0000 7fd7b9a03700 1 -- 172.21.15.27:0/1698111685 reap_dead start 2023-10-06T01:42:18.980 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.971+0000 7fd7bffc7000 1 -- 172.21.15.27:0/1698111685 shutdown_connections 2023-10-06T01:42:18.980 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.971+0000 7fd7bffc7000 1 -- 172.21.15.27:0/1698111685 >> 172.21.15.27:0/1698111685 conn(0x55ba52235120 msgr2=0x55ba52309bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:18.980 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.971+0000 7fd7bffc7000 1 -- 172.21.15.27:0/1698111685 shutdown_connections 2023-10-06T01:42:18.980 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:18.972+0000 7fd7bffc7000 1 -- 172.21.15.27:0/1698111685 wait complete. 2023-10-06T01:42:18.981 INFO:teuthology.orchestra.run.smithi027.stdout:a��alphaparent_b 2023-10-06T01:42:18.981 DEBUG:teuthology.orchestra.run.smithi027:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2023-10-06T01:42:19.075 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2023-10-06T01:42:19.075 INFO:teuthology.orchestra.run.smithi027.stdout: "ino": 1099511628283, 2023-10-06T01:42:19.075 INFO:teuthology.orchestra.run.smithi027.stdout: "ancestors": [ 2023-10-06T01:42:19.076 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:42:19.076 INFO:teuthology.orchestra.run.smithi027.stdout: "dirino": 1099511628284, 2023-10-06T01:42:19.076 INFO:teuthology.orchestra.run.smithi027.stdout: "dname": "alpha", 2023-10-06T01:42:19.076 INFO:teuthology.orchestra.run.smithi027.stdout: "version": 7 2023-10-06T01:42:19.076 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2023-10-06T01:42:19.076 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:42:19.076 INFO:teuthology.orchestra.run.smithi027.stdout: "dirino": 1, 2023-10-06T01:42:19.076 INFO:teuthology.orchestra.run.smithi027.stdout: "dname": "parent_b", 2023-10-06T01:42:19.076 INFO:teuthology.orchestra.run.smithi027.stdout: "version": 30 2023-10-06T01:42:19.076 INFO:teuthology.orchestra.run.smithi027.stdout: } 2023-10-06T01:42:19.076 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2023-10-06T01:42:19.076 INFO:teuthology.orchestra.run.smithi027.stdout: "pool": 7, 2023-10-06T01:42:19.076 INFO:teuthology.orchestra.run.smithi027.stdout: "old_pools": [ 2023-10-06T01:42:19.077 INFO:teuthology.orchestra.run.smithi027.stdout: 6 2023-10-06T01:42:19.077 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2023-10-06T01:42:19.077 INFO:teuthology.orchestra.run.smithi027.stdout:} 2023-10-06T01:42:19.077 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:19.078 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool data_new getxattr 100000001fb.00000000 layout 2023-10-06T01:42:19.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.155+0000 7f3a9bf53000 1 Processor -- start 2023-10-06T01:42:19.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.155+0000 7f3a9bf53000 1 -- start start 2023-10-06T01:42:19.161 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.156+0000 7f3a9bf53000 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x5607d2a2d680 0x5607d2a9e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:19.161 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.156+0000 7f3a9bf53000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5607d2aa2ad0 0x5607d29e74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:19.161 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.156+0000 7f3a9bf53000 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x5607d2aa2ea0 0x5607d2aa1030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:19.161 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.156+0000 7f3a9bf53000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607d2930980 con 0x5607d2aa2ad0 2023-10-06T01:42:19.161 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.156+0000 7f3a9bf53000 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607d2930620 con 0x5607d2a2d680 2023-10-06T01:42:19.161 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.156+0000 7f3a9bf53000 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x5607d2927400 con 0x5607d2aa2ea0 2023-10-06T01:42:19.162 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.156+0000 7f3a9518e700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x5607d2a2d680 0x5607d2a9e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:19.162 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.156+0000 7f3a9498d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5607d2aa2ad0 0x5607d29e74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:19.162 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.156+0000 7f3a9518e700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x5607d2a2d680 0x5607d2a9e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.27:58626/0 (socket says 172.21.15.27:58626) 2023-10-06T01:42:19.162 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.156+0000 7f3a9598f700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x5607d2aa2ea0 0x5607d2aa1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:19.162 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.156+0000 7f3a9518e700 1 -- 172.21.15.27:0/950089857 learned_addr learned my addr 172.21.15.27:0/950089857 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:19.162 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.157+0000 7f3a9498d700 1 -- 172.21.15.27:0/950089857 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x5607d2aa2ea0 msgr2=0x5607d2aa1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:19.162 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.157+0000 7f3a9498d700 1 --2- 172.21.15.27:0/950089857 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x5607d2aa2ea0 0x5607d2aa1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:19.162 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.157+0000 7f3a9498d700 1 -- 172.21.15.27:0/950089857 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x5607d2a2d680 msgr2=0x5607d2a9e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:19.163 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.157+0000 7f3a9498d700 1 --2- 172.21.15.27:0/950089857 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x5607d2a2d680 0x5607d2a9e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:19.163 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.157+0000 7f3a9498d700 1 -- 172.21.15.27:0/950089857 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a8c005d50 con 0x5607d2aa2ad0 2023-10-06T01:42:19.163 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.157+0000 7f3a9498d700 1 --2- 172.21.15.27:0/950089857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5607d2aa2ad0 0x5607d29e74f0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x5607d2a2d040 tx=0x7f3a8c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7cae3fbb35348af server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:19.163 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.157+0000 7f3a877fe700 1 -- 172.21.15.27:0/950089857 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8c01a070 con 0x5607d2aa2ad0 2023-10-06T01:42:19.163 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.157+0000 7f3a877fe700 1 -- 172.21.15.27:0/950089857 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a8c000bc0 con 0x5607d2aa2ad0 2023-10-06T01:42:19.163 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.157+0000 7f3a877fe700 1 -- 172.21.15.27:0/950089857 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8c00bc50 con 0x5607d2aa2ad0 2023-10-06T01:42:19.163 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.157+0000 7f3a9bf53000 1 -- 172.21.15.27:0/950089857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5607d2aa2ad0 msgr2=0x5607d29e74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:19.164 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.157+0000 7f3a9bf53000 1 --2- 172.21.15.27:0/950089857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5607d2aa2ad0 0x5607d29e74f0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x5607d2a2d040 tx=0x7f3a8c014000 comp rx=0 tx=0).stop 2023-10-06T01:42:19.164 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.157+0000 7f3a9bf53000 1 -- 172.21.15.27:0/950089857 shutdown_connections 2023-10-06T01:42:19.164 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.157+0000 7f3a9bf53000 1 --2- 172.21.15.27:0/950089857 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x5607d2aa2ea0 0x5607d2aa1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:19.164 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.157+0000 7f3a9bf53000 1 --2- 172.21.15.27:0/950089857 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x5607d2a2d680 0x5607d2a9e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:19.164 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.157+0000 7f3a9bf53000 1 --2- 172.21.15.27:0/950089857 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5607d2aa2ad0 0x5607d29e74f0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:19.164 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.157+0000 7f3a9bf53000 1 -- 172.21.15.27:0/950089857 >> 172.21.15.27:0/950089857 conn(0x5607d29e7120 msgr2=0x5607d2a2f410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:19.164 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.157+0000 7f3a9bf53000 1 -- 172.21.15.27:0/950089857 shutdown_connections 2023-10-06T01:42:19.164 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.157+0000 7f3a9bf53000 1 -- 172.21.15.27:0/950089857 wait complete. 2023-10-06T01:42:19.165 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.158+0000 7f3a9bf53000 1 Processor -- start 2023-10-06T01:42:19.165 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.158+0000 7f3a9bf53000 1 -- start start 2023-10-06T01:42:19.165 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.159+0000 7f3a9bf53000 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x5607d2a2d680 0x5607d2a2c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:19.165 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.159+0000 7f3a9bf53000 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x5607d2aa2ad0 0x5607d2a9ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:19.165 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.159+0000 7f3a9bf53000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5607d2aa2ea0 0x5607d2a9b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:19.165 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.159+0000 7f3a9bf53000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607d2a9d1c0 con 0x5607d2aa2ea0 2023-10-06T01:42:19.165 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.159+0000 7f3a9bf53000 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607d2a9d460 con 0x5607d2aa2ad0 2023-10-06T01:42:19.165 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.159+0000 7f3a9bf53000 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x5607d2a9c400 con 0x5607d2a2d680 2023-10-06T01:42:19.166 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.159+0000 7f3a9598f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5607d2aa2ea0 0x5607d2a9b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:19.166 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.159+0000 7f3a9518e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x5607d2a2d680 0x5607d2a2c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:19.166 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.159+0000 7f3a9498d700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x5607d2aa2ad0 0x5607d2a9ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:19.166 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.159+0000 7f3a9598f700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5607d2aa2ea0 0x5607d2a9b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59902/0 (socket says 172.21.15.27:59902) 2023-10-06T01:42:19.166 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.159+0000 7f3a9598f700 1 -- 172.21.15.27:0/3237132 learned_addr learned my addr 172.21.15.27:0/3237132 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:19.166 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.159+0000 7f3a9598f700 1 -- 172.21.15.27:0/3237132 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x5607d2a2d680 msgr2=0x5607d2a2c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:19.166 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.159+0000 7f3a9598f700 1 --2- 172.21.15.27:0/3237132 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x5607d2a2d680 0x5607d2a2c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:19.166 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.159+0000 7f3a9598f700 1 -- 172.21.15.27:0/3237132 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x5607d2aa2ad0 msgr2=0x5607d2a9ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:19.167 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.159+0000 7f3a9598f700 1 --2- 172.21.15.27:0/3237132 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x5607d2aa2ad0 0x5607d2a9ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:19.167 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.159+0000 7f3a9598f700 1 -- 172.21.15.27:0/3237132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5607d29591c0 con 0x5607d2aa2ea0 2023-10-06T01:42:19.167 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.159+0000 7f3a9598f700 1 --2- 172.21.15.27:0/3237132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5607d2aa2ea0 0x5607d2a9b740 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f3a9000d490 tx=0x7f3a90006a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:19.167 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.159+0000 7f3a85ffb700 1 -- 172.21.15.27:0/3237132 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a9000dea0 con 0x5607d2aa2ea0 2023-10-06T01:42:19.167 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.159+0000 7f3a85ffb700 1 -- 172.21.15.27:0/3237132 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a90013480 con 0x5607d2aa2ea0 2023-10-06T01:42:19.167 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.159+0000 7f3a85ffb700 1 -- 172.21.15.27:0/3237132 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a90013840 con 0x5607d2aa2ea0 2023-10-06T01:42:19.167 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.159+0000 7f3a9bf53000 1 -- 172.21.15.27:0/3237132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5607d293f3d0 con 0x5607d2aa2ea0 2023-10-06T01:42:19.167 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.160+0000 7f3a9bf53000 1 -- 172.21.15.27:0/3237132 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5607d2932ae0 con 0x5607d2aa2ea0 2023-10-06T01:42:19.169 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.161+0000 7f3a85ffb700 1 -- 172.21.15.27:0/3237132 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3a90013c30 con 0x5607d2aa2ea0 2023-10-06T01:42:19.170 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.161+0000 7f3a85ffb700 1 --2- 172.21.15.27:0/3237132 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3a780658c0 0x7f3a78067d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:19.170 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.161+0000 7f3a85ffb700 1 -- 172.21.15.27:0/3237132 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f3a900835e0 con 0x5607d2aa2ea0 2023-10-06T01:42:19.170 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.161+0000 7f3a9518e700 1 --2- 172.21.15.27:0/3237132 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3a780658c0 0x7f3a78067d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:19.170 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.161+0000 7f3a9bf53000 1 --2- 172.21.15.27:0/3237132 >> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] conn(0x5607d2a2aef0 0x5607d2ad92c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:19.170 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.161+0000 7f3a9bf53000 1 -- 172.21.15.27:0/3237132 --> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] -- osd_op(unknown.0.0:1 7.13 7:ca8fdbba:::100000001fb.00000000:head [getxattr layout in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e34) v8 -- 0x5607d29e7800 con 0x5607d2a2aef0 2023-10-06T01:42:19.170 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.162+0000 7f3a9498d700 1 --2- 172.21.15.27:0/3237132 >> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] conn(0x5607d2a2aef0 0x5607d2ad92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:19.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.162+0000 7f3a9518e700 1 --2- 172.21.15.27:0/3237132 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3a780658c0 0x7f3a78067d70 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3a88005e80 tx=0x7f3a8800c040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:19.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.162+0000 7f3a9498d700 1 --2- 172.21.15.27:0/3237132 >> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] conn(0x5607d2a2aef0 0x5607d2ad92c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:19.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.162+0000 7f3a9498d700 1 -- 172.21.15.27:0/3237132 <== osd.0 v2:172.21.15.27:6800/90644 1 ==== osd_op_reply(1 100000001fb.00000000 [getxattr (30) out=30b] v0'0 uv1 ondisk = 0) v8 ==== 164+0+30 (crc 0 0 0) 0x7f3a8c027040 con 0x5607d2a2aef0 2023-10-06T01:42:19.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.162+0000 7f3a9bf53000 1 -- 172.21.15.27:0/3237132 >> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] conn(0x5607d2a2aef0 msgr2=0x5607d2ad92c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:19.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.162+0000 7f3a9bf53000 1 --2- 172.21.15.27:0/3237132 >> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] conn(0x5607d2a2aef0 0x5607d2ad92c0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:19.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.162+0000 7f3a9bf53000 1 -- 172.21.15.27:0/3237132 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3a780658c0 msgr2=0x7f3a78067d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:19.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.162+0000 7f3a9bf53000 1 --2- 172.21.15.27:0/3237132 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3a780658c0 0x7f3a78067d70 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3a88005e80 tx=0x7f3a8800c040 comp rx=0 tx=0).stop 2023-10-06T01:42:19.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.162+0000 7f3a9bf53000 1 -- 172.21.15.27:0/3237132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5607d2aa2ea0 msgr2=0x5607d2a9b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:19.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.162+0000 7f3a9bf53000 1 --2- 172.21.15.27:0/3237132 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x5607d2aa2ea0 0x5607d2a9b740 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f3a9000d490 tx=0x7f3a90006a80 comp rx=0 tx=0).stop 2023-10-06T01:42:19.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.162+0000 7f3a9598f700 1 -- 172.21.15.27:0/3237132 reap_dead start 2023-10-06T01:42:19.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.162+0000 7f3a9bf53000 1 -- 172.21.15.27:0/3237132 shutdown_connections 2023-10-06T01:42:19.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.162+0000 7f3a9bf53000 1 -- 172.21.15.27:0/3237132 >> 172.21.15.27:0/3237132 conn(0x5607d29e7120 msgr2=0x5607d2a2f410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:19.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.162+0000 7f3a9bf53000 1 -- 172.21.15.27:0/3237132 shutdown_connections 2023-10-06T01:42:19.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.162+0000 7f3a9bf53000 1 -- 172.21.15.27:0/3237132 wait complete. 2023-10-06T01:42:19.173 INFO:teuthology.orchestra.run.smithi027.stdout:@@ 2023-10-06T01:42:19.174 DEBUG:teuthology.orchestra.run.smithi027:> ceph-dencoder type file_layout_t import - decode dump_json 2023-10-06T01:42:19.267 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2023-10-06T01:42:19.267 INFO:teuthology.orchestra.run.smithi027.stdout: "stripe_unit": 4194304, 2023-10-06T01:42:19.268 INFO:teuthology.orchestra.run.smithi027.stdout: "stripe_count": 1, 2023-10-06T01:42:19.268 INFO:teuthology.orchestra.run.smithi027.stdout: "object_size": 4194304, 2023-10-06T01:42:19.268 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_id": 7, 2023-10-06T01:42:19.268 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_ns": "" 2023-10-06T01:42:19.268 INFO:teuthology.orchestra.run.smithi027.stdout:} 2023-10-06T01:42:19.268 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:19.269 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T01:42:19.270 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parent_c) 2023-10-06T01:42:19.326 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T01:42:19.327 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec mv parent_b/alpha parent_c/alpha) 2023-10-06T01:42:19.386 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T01:42:19.545 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.543+0000 7faefe0f5700 1 Processor -- start 2023-10-06T01:42:19.545 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.544+0000 7faefe0f5700 1 -- start start 2023-10-06T01:42:19.547 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.544+0000 7faefe0f5700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7faef815d930 0x7faef80d0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:19.547 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.544+0000 7faefe0f5700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7faef81591b0 0x7faef8159600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:19.547 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.544+0000 7faefe0f5700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faef80b2ad0 con 0x7faef815d560 2023-10-06T01:42:19.547 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.544+0000 7faefe0f5700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faef80b4160 con 0x7faef815d930 2023-10-06T01:42:19.547 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.544+0000 7faefe0f5700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faef80b2230 con 0x7faef81591b0 2023-10-06T01:42:19.547 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.544+0000 7faef6ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7faef815d930 0x7faef80d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:19.547 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.545+0000 7faef6ffd700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7faef815d930 0x7faef80d0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59908/0 (socket says 172.21.15.27:59908) 2023-10-06T01:42:19.547 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.545+0000 7faef6ffd700 1 -- 172.21.15.27:0/1118506912 learned_addr learned my addr 172.21.15.27:0/1118506912 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:19.548 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.545+0000 7faef7fff700 1 --2- 172.21.15.27:0/1118506912 >> v2:172.21.15.124:3300/0 conn(0x7faef81591b0 0x7faef8159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:19.548 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.545+0000 7faef6ffd700 1 -- 172.21.15.27:0/1118506912 >> v2:172.21.15.124:3300/0 conn(0x7faef81591b0 msgr2=0x7faef8159600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:19.548 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.545+0000 7faef6ffd700 1 --2- 172.21.15.27:0/1118506912 >> v2:172.21.15.124:3300/0 conn(0x7faef81591b0 0x7faef8159600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:19.548 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.545+0000 7faef6ffd700 1 -- 172.21.15.27:0/1118506912 >> v1:172.21.15.124:6789/0 conn(0x7faef815d560 legacy=0x7faef80b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:19.548 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.545+0000 7faef6ffd700 1 -- 172.21.15.27:0/1118506912 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef80b2510 con 0x7faef815d930 2023-10-06T01:42:19.548 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.545+0000 7faef6ffd700 1 --2- 172.21.15.27:0/1118506912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faef815d930 0x7faef80d0030 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7faef80b3af0 tx=0x7faeec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1706ddfd2b953d0d server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:19.548 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.545+0000 7faef67fc700 1 -- 172.21.15.27:0/1118506912 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeec009e70 con 0x7faef815d930 2023-10-06T01:42:19.548 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.545+0000 7faef67fc700 1 -- 172.21.15.27:0/1118506912 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeec00b860 con 0x7faef815d930 2023-10-06T01:42:19.548 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.545+0000 7faef67fc700 1 -- 172.21.15.27:0/1118506912 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeec00bc00 con 0x7faef815d930 2023-10-06T01:42:19.548 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.546+0000 7faefe0f5700 1 -- 172.21.15.27:0/1118506912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faef815d930 msgr2=0x7faef80d0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:19.548 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.546+0000 7faefe0f5700 1 --2- 172.21.15.27:0/1118506912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faef815d930 0x7faef80d0030 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7faef80b3af0 tx=0x7faeec009250 comp rx=0 tx=0).stop 2023-10-06T01:42:19.548 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.546+0000 7faefe0f5700 1 -- 172.21.15.27:0/1118506912 shutdown_connections 2023-10-06T01:42:19.549 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.546+0000 7faefe0f5700 1 --2- 172.21.15.27:0/1118506912 >> v2:172.21.15.124:3300/0 conn(0x7faef81591b0 0x7faef8159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:19.549 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.546+0000 7faefe0f5700 1 --2- 172.21.15.27:0/1118506912 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faef815d930 0x7faef80d0030 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:19.549 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.546+0000 7faefe0f5700 1 -- 172.21.15.27:0/1118506912 >> 172.21.15.27:0/1118506912 conn(0x7faef80d1180 msgr2=0x7faef8069640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:19.549 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.546+0000 7faefe0f5700 1 -- 172.21.15.27:0/1118506912 shutdown_connections 2023-10-06T01:42:19.549 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.546+0000 7faefe0f5700 1 -- 172.21.15.27:0/1118506912 wait complete. 2023-10-06T01:42:19.551 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.547+0000 7faefe0f5700 1 Processor -- start 2023-10-06T01:42:19.551 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.547+0000 7faefe0f5700 1 -- start start 2023-10-06T01:42:19.551 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.548+0000 7faefe0f5700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faef81591b0 0x7faef8151560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:19.551 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.548+0000 7faefe0f5700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7faef815d560 0x7faef8153950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:19.551 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.548+0000 7faefe0f5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faef815d930 0x7faef816a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:19.551 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.548+0000 7faefe0f5700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef8167b30 con 0x7faef815d930 2023-10-06T01:42:19.552 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.548+0000 7faefe0f5700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef815a9e0 con 0x7faef81591b0 2023-10-06T01:42:19.552 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.548+0000 7faefe0f5700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faef8153e90 con 0x7faef815d560 2023-10-06T01:42:19.552 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.548+0000 7faef7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faef815d930 0x7faef816a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:19.552 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.548+0000 7faef7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faef815d930 0x7faef816a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59918/0 (socket says 172.21.15.27:59918) 2023-10-06T01:42:19.552 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.548+0000 7faef7fff700 1 -- 172.21.15.27:0/2331092963 learned_addr learned my addr 172.21.15.27:0/2331092963 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:19.552 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.548+0000 7faef6ffd700 1 --2- 172.21.15.27:0/2331092963 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7faef815d560 0x7faef8153950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:19.552 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.548+0000 7faef77fe700 1 --2- 172.21.15.27:0/2331092963 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faef81591b0 0x7faef8151560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:19.552 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.548+0000 7faef7fff700 1 -- 172.21.15.27:0/2331092963 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7faef815d560 msgr2=0x7faef8153950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:19.552 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.548+0000 7faef7fff700 1 --2- 172.21.15.27:0/2331092963 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7faef815d560 0x7faef8153950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:19.552 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.548+0000 7faef7fff700 1 -- 172.21.15.27:0/2331092963 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faef81591b0 msgr2=0x7faef8151560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:19.552 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.548+0000 7faef7fff700 1 --2- 172.21.15.27:0/2331092963 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faef81591b0 0x7faef8151560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:19.552 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.548+0000 7faef7fff700 1 -- 172.21.15.27:0/2331092963 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeec0094f0 con 0x7faef815d930 2023-10-06T01:42:19.553 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.548+0000 7faef7fff700 1 --2- 172.21.15.27:0/2331092963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faef815d930 0x7faef816a0c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7faef0009ec0 tx=0x7faef000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:19.553 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.548+0000 7faedf7fe700 1 -- 172.21.15.27:0/2331092963 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef000cb70 con 0x7faef815d930 2023-10-06T01:42:19.553 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.549+0000 7faedf7fe700 1 -- 172.21.15.27:0/2331092963 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faef000f040 con 0x7faef815d930 2023-10-06T01:42:19.553 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.549+0000 7faefe0f5700 1 -- 172.21.15.27:0/2331092963 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faef814e4a0 con 0x7faef815d930 2023-10-06T01:42:19.553 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.549+0000 7faedf7fe700 1 -- 172.21.15.27:0/2331092963 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef0014480 con 0x7faef815d930 2023-10-06T01:42:19.553 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.549+0000 7faefe0f5700 1 -- 172.21.15.27:0/2331092963 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faef81543f0 con 0x7faef815d930 2023-10-06T01:42:19.553 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.550+0000 7faefe0f5700 1 -- 172.21.15.27:0/2331092963 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faef8064050 con 0x7faef815d930 2023-10-06T01:42:19.553 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.550+0000 7faedf7fe700 1 -- 172.21.15.27:0/2331092963 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7faef0014830 con 0x7faef815d930 2023-10-06T01:42:19.553 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.550+0000 7faedf7fe700 1 --2- 172.21.15.27:0/2331092963 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7faee4066f30 0x7faee40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:19.554 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.550+0000 7faedf7fe700 1 -- 172.21.15.27:0/2331092963 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6453+0+0 (secure 0 0 0) 0x7faef00851d0 con 0x7faef815d930 2023-10-06T01:42:19.554 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.550+0000 7faef77fe700 1 --2- 172.21.15.27:0/2331092963 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7faee4066f30 0x7faee40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:19.554 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.551+0000 7faef77fe700 1 --2- 172.21.15.27:0/2331092963 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7faee4066f30 0x7faee40693e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7faef81555b0 tx=0x7faee8009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:19.556 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.555+0000 7faedf7fe700 1 -- 172.21.15.27:0/2331092963 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faef0052150 con 0x7faef815d930 2023-10-06T01:42:19.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.795+0000 7faefe0f5700 1 -- 172.21.15.27:0/2331092963 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faef80b27f0 con 0x7faef815d930 2023-10-06T01:42:19.796 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.795+0000 7faedf7fe700 1 -- 172.21.15.27:0/2331092963 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4599 (secure 0 0 0) 0x7faef0055770 con 0x7faef815d930 2023-10-06T01:42:19.797 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:19.797 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4870,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6836/1043044736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":1043044736},{"type":"v1","addr":"172.21.15.124:6836","nonce":1043044736}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4962,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/1207987117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1207987117},{"type":"v1","addr":"172.21.15.27:6834","nonce":1207987117}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4965,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/943261840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":943261840},{"type":"v1","addr":"172.21.15.27:6835","nonce":943261840}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T01:42:03.036784+0000","modified":"2023-10-06T01:42:17.363571+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4974},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4974":{"gid":4974,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.124:6837/325247112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6835","nonce":325247112},{"type":"v1","addr":"172.21.15.124:6837","nonce":325247112}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6,7],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-10-06T01:42:19.799 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.797+0000 7faefe0f5700 1 -- 172.21.15.27:0/2331092963 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7faee4066f30 msgr2=0x7faee40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:19.799 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.797+0000 7faefe0f5700 1 --2- 172.21.15.27:0/2331092963 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7faee4066f30 0x7faee40693e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7faef81555b0 tx=0x7faee8009040 comp rx=0 tx=0).stop 2023-10-06T01:42:19.799 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.797+0000 7faefe0f5700 1 -- 172.21.15.27:0/2331092963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faef815d930 msgr2=0x7faef816a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:19.799 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.797+0000 7faefe0f5700 1 --2- 172.21.15.27:0/2331092963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faef815d930 0x7faef816a0c0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7faef0009ec0 tx=0x7faef000b040 comp rx=0 tx=0).stop 2023-10-06T01:42:19.799 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.797+0000 7faefe0f5700 1 -- 172.21.15.27:0/2331092963 shutdown_connections 2023-10-06T01:42:19.799 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.797+0000 7faefe0f5700 1 --2- 172.21.15.27:0/2331092963 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7faee4066f30 0x7faee40693e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:19.799 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.797+0000 7faefe0f5700 1 --2- 172.21.15.27:0/2331092963 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7faef815d930 0x7faef816a0c0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:19.799 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.797+0000 7faefe0f5700 1 --2- 172.21.15.27:0/2331092963 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7faef81591b0 0x7faef8151560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:19.800 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.797+0000 7faefe0f5700 1 --2- 172.21.15.27:0/2331092963 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7faef815d560 0x7faef8153950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:19.800 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.797+0000 7faefe0f5700 1 -- 172.21.15.27:0/2331092963 >> 172.21.15.27:0/2331092963 conn(0x7faef80d1180 msgr2=0x7faef8068c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:19.800 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.797+0000 7faefe0f5700 1 -- 172.21.15.27:0/2331092963 shutdown_connections 2023-10-06T01:42:19.800 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.797+0000 7faefe0f5700 1 -- 172.21.15.27:0/2331092963 wait complete. 2023-10-06T01:42:19.800 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 16 2023-10-06T01:42:19.811 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json flush journal 2023-10-06T01:42:19.952 INFO:teuthology.orchestra.run.smithi124.stdout:{"message":"","return_code":0} 2023-10-06T01:42:19.952 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2023-10-06T01:42:19.953 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001fb.00000000 parent 2023-10-06T01:42:19.992 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.991+0000 7f4d71a5d000 1 Processor -- start 2023-10-06T01:42:19.992 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.991+0000 7f4d71a5d000 1 -- start start 2023-10-06T01:42:19.997 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.991+0000 7f4d71a5d000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557a888f0cb0 0x557a889bce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:19.997 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.991+0000 7f4d71a5d000 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x557a889186d0 0x557a889379a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:19.997 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.991+0000 7f4d71a5d000 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x557a889c3290 0x557a8891c980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:19.997 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.991+0000 7f4d71a5d000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a88839620 con 0x557a888f0cb0 2023-10-06T01:42:19.997 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.991+0000 7f4d71a5d000 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a88830400 con 0x557a889186d0 2023-10-06T01:42:19.997 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.991+0000 7f4d71a5d000 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x557a889c05b0 con 0x557a889c3290 2023-10-06T01:42:19.997 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.991+0000 7f4d6ac98700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557a888f0cb0 0x557a889bce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:19.998 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.992+0000 7f4d6ac98700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557a888f0cb0 0x557a889bce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59932/0 (socket says 172.21.15.27:59932) 2023-10-06T01:42:19.998 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.992+0000 7f4d6ac98700 1 -- 172.21.15.27:0/2573512505 learned_addr learned my addr 172.21.15.27:0/2573512505 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:19.998 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.992+0000 7f4d63fff700 1 --2- 172.21.15.27:0/2573512505 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x557a889186d0 0x557a889379a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:19.998 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.992+0000 7f4d6b499700 1 --2- 172.21.15.27:0/2573512505 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x557a889c3290 0x557a8891c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:19.998 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.992+0000 7f4d6ac98700 1 -- 172.21.15.27:0/2573512505 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x557a889c3290 msgr2=0x557a8891c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:19.998 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.992+0000 7f4d6ac98700 1 --2- 172.21.15.27:0/2573512505 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x557a889c3290 0x557a8891c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:19.998 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.992+0000 7f4d6ac98700 1 -- 172.21.15.27:0/2573512505 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x557a889186d0 msgr2=0x557a889379a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:19.999 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.992+0000 7f4d6ac98700 1 --2- 172.21.15.27:0/2573512505 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x557a889186d0 0x557a889379a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:19.999 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.992+0000 7f4d6ac98700 1 -- 172.21.15.27:0/2573512505 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d5c005db0 con 0x557a888f0cb0 2023-10-06T01:42:19.999 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.992+0000 7f4d6ac98700 1 --2- 172.21.15.27:0/2573512505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557a888f0cb0 0x557a889bce00 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c008410 tx=0x7f4d5c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=209bf908bd85fab4 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:19.999 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.992+0000 7f4d69c96700 1 -- 172.21.15.27:0/2573512505 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5c000bc0 con 0x557a888f0cb0 2023-10-06T01:42:19.999 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.992+0000 7f4d69c96700 1 -- 172.21.15.27:0/2573512505 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d5c008a50 con 0x557a888f0cb0 2023-10-06T01:42:19.999 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.992+0000 7f4d69c96700 1 -- 172.21.15.27:0/2573512505 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5c000d20 con 0x557a888f0cb0 2023-10-06T01:42:19.999 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.992+0000 7f4d71a5d000 1 -- 172.21.15.27:0/2573512505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557a888f0cb0 msgr2=0x557a889bce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:19.999 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.992+0000 7f4d71a5d000 1 --2- 172.21.15.27:0/2573512505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557a888f0cb0 0x557a889bce00 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c008410 tx=0x7f4d5c016000 comp rx=0 tx=0).stop 2023-10-06T01:42:20.000 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.993+0000 7f4d71a5d000 1 -- 172.21.15.27:0/2573512505 shutdown_connections 2023-10-06T01:42:20.000 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.993+0000 7f4d71a5d000 1 --2- 172.21.15.27:0/2573512505 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x557a889c3290 0x557a8891c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:20.000 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.993+0000 7f4d71a5d000 1 --2- 172.21.15.27:0/2573512505 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557a888f0cb0 0x557a889bce00 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:20.000 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.993+0000 7f4d71a5d000 1 --2- 172.21.15.27:0/2573512505 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x557a889186d0 0x557a889379a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:20.000 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.993+0000 7f4d71a5d000 1 -- 172.21.15.27:0/2573512505 >> 172.21.15.27:0/2573512505 conn(0x557a888ef8a0 msgr2=0x557a8891abf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:20.000 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.993+0000 7f4d71a5d000 1 -- 172.21.15.27:0/2573512505 shutdown_connections 2023-10-06T01:42:20.000 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.993+0000 7f4d71a5d000 1 -- 172.21.15.27:0/2573512505 wait complete. 2023-10-06T01:42:20.000 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.993+0000 7f4d71a5d000 1 Processor -- start 2023-10-06T01:42:20.001 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.993+0000 7f4d71a5d000 1 -- start start 2023-10-06T01:42:20.001 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.994+0000 7f4d71a5d000 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x557a888f0cb0 0x557a8892ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:20.001 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.994+0000 7f4d71a5d000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557a889186d0 0x557a8891c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:20.001 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.994+0000 7f4d71a5d000 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x557a889c3290 0x557a8891ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:20.001 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.994+0000 7f4d71a5d000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a889d5390 con 0x557a889186d0 2023-10-06T01:42:20.001 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.994+0000 7f4d71a5d000 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x557a889d55f0 con 0x557a889c3290 2023-10-06T01:42:20.001 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.994+0000 7f4d71a5d000 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x557a889d5890 con 0x557a888f0cb0 2023-10-06T01:42:20.001 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.994+0000 7f4d63fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557a889186d0 0x557a8891c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:20.002 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.994+0000 7f4d63fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557a889186d0 0x557a8891c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59940/0 (socket says 172.21.15.27:59940) 2023-10-06T01:42:20.002 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.994+0000 7f4d63fff700 1 -- 172.21.15.27:0/4203524950 learned_addr learned my addr 172.21.15.27:0/4203524950 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:20.002 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.994+0000 7f4d6ac98700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x557a888f0cb0 0x557a8892ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:20.002 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.994+0000 7f4d6b499700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x557a889c3290 0x557a8891ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:20.002 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.994+0000 7f4d63fff700 1 -- 172.21.15.27:0/4203524950 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x557a888f0cb0 msgr2=0x557a8892ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:20.002 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.994+0000 7f4d63fff700 1 --2- 172.21.15.27:0/4203524950 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x557a888f0cb0 0x557a8892ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:20.002 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.994+0000 7f4d63fff700 1 -- 172.21.15.27:0/4203524950 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x557a889c3290 msgr2=0x557a8891ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:20.003 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.994+0000 7f4d63fff700 1 --2- 172.21.15.27:0/4203524950 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x557a889c3290 0x557a8891ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:20.003 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.994+0000 7f4d63fff700 1 -- 172.21.15.27:0/4203524950 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557a8883b750 con 0x557a889186d0 2023-10-06T01:42:20.003 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.994+0000 7f4d6ac98700 1 --2- 172.21.15.27:0/4203524950 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x557a888f0cb0 0x557a8892ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:42:20.003 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.995+0000 7f4d63fff700 1 --2- 172.21.15.27:0/4203524950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557a889186d0 0x557a8891c4e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f4d5400ab10 tx=0x7f4d5400e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:20.003 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.995+0000 7f4d637fe700 1 -- 172.21.15.27:0/4203524950 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d54011040 con 0x557a889186d0 2023-10-06T01:42:20.003 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.995+0000 7f4d637fe700 1 -- 172.21.15.27:0/4203524950 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d5400b450 con 0x557a889186d0 2023-10-06T01:42:20.003 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.995+0000 7f4d71a5d000 1 -- 172.21.15.27:0/4203524950 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x557a888621c0 con 0x557a889186d0 2023-10-06T01:42:20.003 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.995+0000 7f4d637fe700 1 -- 172.21.15.27:0/4203524950 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d54008cd0 con 0x557a889186d0 2023-10-06T01:42:20.004 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.995+0000 7f4d71a5d000 1 -- 172.21.15.27:0/4203524950 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557a888483d0 con 0x557a889186d0 2023-10-06T01:42:20.006 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.996+0000 7f4d637fe700 1 -- 172.21.15.27:0/4203524950 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f4d54016690 con 0x557a889186d0 2023-10-06T01:42:20.006 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.996+0000 7f4d637fe700 1 --2- 172.21.15.27:0/4203524950 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4d4c065890 0x7f4d4c067d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:20.006 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.997+0000 7f4d637fe700 1 -- 172.21.15.27:0/4203524950 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f4d54083680 con 0x557a889186d0 2023-10-06T01:42:20.006 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.997+0000 7f4d6ac98700 1 --2- 172.21.15.27:0/4203524950 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4d4c065890 0x7f4d4c067d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:20.006 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.997+0000 7f4d71a5d000 1 --2- 172.21.15.27:0/4203524950 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x557a889d5fd0 0x557a889d63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:20.006 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.997+0000 7f4d71a5d000 1 -- 172.21.15.27:0/4203524950 --> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] -- osd_op(unknown.0.0:1 6.13 6:ca8fdbba:::100000001fb.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e34) v8 -- 0x557a889d6910 con 0x557a889d5fd0 2023-10-06T01:42:20.007 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.997+0000 7f4d6b499700 1 --2- 172.21.15.27:0/4203524950 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x557a889d5fd0 0x557a889d63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:20.007 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.997+0000 7f4d6ac98700 1 --2- 172.21.15.27:0/4203524950 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4d4c065890 0x7f4d4c067d40 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c008410 tx=0x7f4d5c01c000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:20.007 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.997+0000 7f4d6b499700 1 --2- 172.21.15.27:0/4203524950 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x557a889d5fd0 0x557a889d63d0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:20.007 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.998+0000 7f4d6b499700 1 -- 172.21.15.27:0/4203524950 <== osd.3 v2:172.21.15.27:6824/90668 1 ==== osd_op_reply(1 100000001fb.00000000 [getxattr (103) out=103b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+103 (crc 0 0 0) 0x7f4d6400cc20 con 0x557a889d5fd0 2023-10-06T01:42:20.007 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.998+0000 7f4d71a5d000 1 -- 172.21.15.27:0/4203524950 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x557a889d5fd0 msgr2=0x557a889d63d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:20.007 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.998+0000 7f4d71a5d000 1 --2- 172.21.15.27:0/4203524950 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x557a889d5fd0 0x557a889d63d0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:20.007 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.998+0000 7f4d71a5d000 1 -- 172.21.15.27:0/4203524950 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4d4c065890 msgr2=0x7f4d4c067d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:20.007 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.998+0000 7f4d71a5d000 1 --2- 172.21.15.27:0/4203524950 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4d4c065890 0x7f4d4c067d40 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4d5c008410 tx=0x7f4d5c01c000 comp rx=0 tx=0).stop 2023-10-06T01:42:20.008 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.998+0000 7f4d71a5d000 1 -- 172.21.15.27:0/4203524950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557a889186d0 msgr2=0x557a8891c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:20.008 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.998+0000 7f4d71a5d000 1 --2- 172.21.15.27:0/4203524950 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x557a889186d0 0x557a8891c4e0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f4d5400ab10 tx=0x7f4d5400e040 comp rx=0 tx=0).stop 2023-10-06T01:42:20.008 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.998+0000 7f4d6b499700 1 -- 172.21.15.27:0/4203524950 reap_dead start 2023-10-06T01:42:20.008 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.998+0000 7f4d71a5d000 1 -- 172.21.15.27:0/4203524950 shutdown_connections 2023-10-06T01:42:20.008 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.998+0000 7f4d71a5d000 1 -- 172.21.15.27:0/4203524950 >> 172.21.15.27:0/4203524950 conn(0x557a888ef8a0 msgr2=0x557a889be6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:20.008 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.998+0000 7f4d71a5d000 1 -- 172.21.15.27:0/4203524950 shutdown_connections 2023-10-06T01:42:20.008 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:19.998+0000 7f4d71a5d000 1 -- 172.21.15.27:0/4203524950 wait complete. 2023-10-06T01:42:20.009 INFO:teuthology.orchestra.run.smithi027.stdout:a��alphaparent_b 2023-10-06T01:42:20.009 DEBUG:teuthology.orchestra.run.smithi027:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2023-10-06T01:42:20.103 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2023-10-06T01:42:20.103 INFO:teuthology.orchestra.run.smithi027.stdout: "ino": 1099511628283, 2023-10-06T01:42:20.103 INFO:teuthology.orchestra.run.smithi027.stdout: "ancestors": [ 2023-10-06T01:42:20.103 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:42:20.103 INFO:teuthology.orchestra.run.smithi027.stdout: "dirino": 1099511628284, 2023-10-06T01:42:20.103 INFO:teuthology.orchestra.run.smithi027.stdout: "dname": "alpha", 2023-10-06T01:42:20.104 INFO:teuthology.orchestra.run.smithi027.stdout: "version": 7 2023-10-06T01:42:20.104 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2023-10-06T01:42:20.104 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:42:20.104 INFO:teuthology.orchestra.run.smithi027.stdout: "dirino": 1, 2023-10-06T01:42:20.104 INFO:teuthology.orchestra.run.smithi027.stdout: "dname": "parent_b", 2023-10-06T01:42:20.104 INFO:teuthology.orchestra.run.smithi027.stdout: "version": 30 2023-10-06T01:42:20.104 INFO:teuthology.orchestra.run.smithi027.stdout: } 2023-10-06T01:42:20.104 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2023-10-06T01:42:20.104 INFO:teuthology.orchestra.run.smithi027.stdout: "pool": 7, 2023-10-06T01:42:20.104 INFO:teuthology.orchestra.run.smithi027.stdout: "old_pools": [ 2023-10-06T01:42:20.105 INFO:teuthology.orchestra.run.smithi027.stdout: 6 2023-10-06T01:42:20.105 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2023-10-06T01:42:20.105 INFO:teuthology.orchestra.run.smithi027.stdout:} 2023-10-06T01:42:20.105 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:20.106 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool data_new getxattr 100000001fb.00000000 parent 2023-10-06T01:42:20.186 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.184+0000 7f62456c7000 1 Processor -- start 2023-10-06T01:42:20.186 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.184+0000 7f62456c7000 1 -- start start 2023-10-06T01:42:20.190 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.185+0000 7f62456c7000 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55bfec13f690 0x55bfec1822d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:20.190 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.185+0000 7f62456c7000 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55bfec1852d0 0x55bfec1804c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:20.190 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.185+0000 7f62456c7000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bfec186890 0x55bfec15f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:20.191 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.185+0000 7f62456c7000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfec087980 con 0x55bfec186890 2023-10-06T01:42:20.191 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.185+0000 7f62456c7000 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfec087620 con 0x55bfec13f690 2023-10-06T01:42:20.191 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.185+0000 7f62456c7000 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bfec07e400 con 0x55bfec1852d0 2023-10-06T01:42:20.191 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.185+0000 7f623f103700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bfec186890 0x55bfec15f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:20.191 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.185+0000 7f623f103700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bfec186890 0x55bfec15f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59956/0 (socket says 172.21.15.27:59956) 2023-10-06T01:42:20.191 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.185+0000 7f623f103700 1 -- 172.21.15.27:0/4006201237 learned_addr learned my addr 172.21.15.27:0/4006201237 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:20.191 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.185+0000 7f623e101700 1 --2- 172.21.15.27:0/4006201237 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55bfec1852d0 0x55bfec1804c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:20.191 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.185+0000 7f623e902700 1 --2- 172.21.15.27:0/4006201237 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55bfec13f690 0x55bfec1822d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:20.191 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.185+0000 7f623f103700 1 -- 172.21.15.27:0/4006201237 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55bfec1852d0 msgr2=0x55bfec1804c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:20.191 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.185+0000 7f623f103700 1 --2- 172.21.15.27:0/4006201237 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55bfec1852d0 0x55bfec1804c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:20.191 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.185+0000 7f623f103700 1 -- 172.21.15.27:0/4006201237 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55bfec13f690 msgr2=0x55bfec1822d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:20.192 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.185+0000 7f623f103700 1 --2- 172.21.15.27:0/4006201237 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55bfec13f690 0x55bfec1822d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:20.192 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.185+0000 7f623f103700 1 -- 172.21.15.27:0/4006201237 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6238005cf0 con 0x55bfec186890 2023-10-06T01:42:20.192 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.185+0000 7f623e101700 1 --2- 172.21.15.27:0/4006201237 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55bfec1852d0 0x55bfec1804c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:42:20.192 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.186+0000 7f623f103700 1 --2- 172.21.15.27:0/4006201237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bfec186890 0x55bfec15f3d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f623000c4f0 tx=0x7f6230007040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a882aed59565b35 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:20.192 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.186+0000 7f623d0ff700 1 -- 172.21.15.27:0/4006201237 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62300115f0 con 0x55bfec186890 2023-10-06T01:42:20.192 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.186+0000 7f623d0ff700 1 -- 172.21.15.27:0/4006201237 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6230011750 con 0x55bfec186890 2023-10-06T01:42:20.192 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.186+0000 7f623d0ff700 1 -- 172.21.15.27:0/4006201237 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6230011af0 con 0x55bfec186890 2023-10-06T01:42:20.192 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.186+0000 7f62456c7000 1 -- 172.21.15.27:0/4006201237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bfec186890 msgr2=0x55bfec15f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:20.192 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.186+0000 7f62456c7000 1 --2- 172.21.15.27:0/4006201237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bfec186890 0x55bfec15f3d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f623000c4f0 tx=0x7f6230007040 comp rx=0 tx=0).stop 2023-10-06T01:42:20.192 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.186+0000 7f62456c7000 1 -- 172.21.15.27:0/4006201237 shutdown_connections 2023-10-06T01:42:20.192 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.186+0000 7f62456c7000 1 --2- 172.21.15.27:0/4006201237 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bfec186890 0x55bfec15f3d0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:20.192 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.186+0000 7f62456c7000 1 --2- 172.21.15.27:0/4006201237 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55bfec13f690 0x55bfec1822d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:20.193 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.186+0000 7f62456c7000 1 --2- 172.21.15.27:0/4006201237 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55bfec1852d0 0x55bfec1804c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:20.193 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.186+0000 7f62456c7000 1 -- 172.21.15.27:0/4006201237 >> 172.21.15.27:0/4006201237 conn(0x55bfec16c7b0 msgr2=0x55bfec183f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:20.193 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.186+0000 7f62456c7000 1 -- 172.21.15.27:0/4006201237 shutdown_connections 2023-10-06T01:42:20.193 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.186+0000 7f62456c7000 1 -- 172.21.15.27:0/4006201237 wait complete. 2023-10-06T01:42:20.193 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.187+0000 7f62456c7000 1 Processor -- start 2023-10-06T01:42:20.193 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.187+0000 7f62456c7000 1 -- start start 2023-10-06T01:42:20.193 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.188+0000 7f62456c7000 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55bfec13f690 0x55bfec20b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:20.193 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.188+0000 7f62456c7000 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55bfec1852d0 0x55bfec2161a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:20.193 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.188+0000 7f62456c7000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bfec186890 0x55bfec17e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:20.193 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.188+0000 7f62456c7000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfec20b900 con 0x55bfec186890 2023-10-06T01:42:20.193 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.188+0000 7f62456c7000 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfec211410 con 0x55bfec13f690 2023-10-06T01:42:20.193 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.188+0000 7f62456c7000 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bfec213530 con 0x55bfec1852d0 2023-10-06T01:42:20.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.188+0000 7f623f103700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bfec186890 0x55bfec17e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:20.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.188+0000 7f623f103700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bfec186890 0x55bfec17e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59962/0 (socket says 172.21.15.27:59962) 2023-10-06T01:42:20.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.188+0000 7f623f103700 1 -- 172.21.15.27:0/2946005087 learned_addr learned my addr 172.21.15.27:0/2946005087 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:20.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.188+0000 7f623e902700 1 --2- 172.21.15.27:0/2946005087 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55bfec13f690 0x55bfec20b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:20.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.188+0000 7f623e101700 1 --2- 172.21.15.27:0/2946005087 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55bfec1852d0 0x55bfec2161a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:20.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.188+0000 7f623f103700 1 -- 172.21.15.27:0/2946005087 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55bfec1852d0 msgr2=0x55bfec2161a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:20.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.188+0000 7f623f103700 1 --2- 172.21.15.27:0/2946005087 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55bfec1852d0 0x55bfec2161a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:20.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.188+0000 7f623f103700 1 -- 172.21.15.27:0/2946005087 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55bfec13f690 msgr2=0x55bfec20b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:20.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.188+0000 7f623f103700 1 --2- 172.21.15.27:0/2946005087 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55bfec13f690 0x55bfec20b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:20.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.188+0000 7f623f103700 1 -- 172.21.15.27:0/2946005087 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f623000d040 con 0x55bfec186890 2023-10-06T01:42:20.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.188+0000 7f623e902700 1 --2- 172.21.15.27:0/2946005087 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55bfec13f690 0x55bfec20b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:42:20.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.188+0000 7f623f103700 1 --2- 172.21.15.27:0/2946005087 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bfec186890 0x55bfec17e960 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f6230003000 tx=0x7f6230018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:20.195 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.188+0000 7f62377fe700 1 -- 172.21.15.27:0/2946005087 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6230012b10 con 0x55bfec186890 2023-10-06T01:42:20.195 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.188+0000 7f62377fe700 1 -- 172.21.15.27:0/2946005087 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6230021070 con 0x55bfec186890 2023-10-06T01:42:20.195 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.189+0000 7f62377fe700 1 -- 172.21.15.27:0/2946005087 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6230026470 con 0x55bfec186890 2023-10-06T01:42:20.195 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.189+0000 7f62456c7000 1 -- 172.21.15.27:0/2946005087 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55bfec0963d0 con 0x55bfec186890 2023-10-06T01:42:20.195 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.189+0000 7f62456c7000 1 -- 172.21.15.27:0/2946005087 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bfec089ae0 con 0x55bfec186890 2023-10-06T01:42:20.197 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.190+0000 7f62377fe700 1 -- 172.21.15.27:0/2946005087 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6230023070 con 0x55bfec186890 2023-10-06T01:42:20.197 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.190+0000 7f62377fe700 1 --2- 172.21.15.27:0/2946005087 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6220065890 0x7f6220067d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:20.197 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.190+0000 7f62377fe700 1 -- 172.21.15.27:0/2946005087 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f6230085800 con 0x55bfec186890 2023-10-06T01:42:20.197 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.190+0000 7f623e902700 1 --2- 172.21.15.27:0/2946005087 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6220065890 0x7f6220067d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:20.197 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.190+0000 7f62456c7000 1 --2- 172.21.15.27:0/2946005087 >> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] conn(0x55bfec2166e0 0x55bfec214450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:20.197 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.190+0000 7f62456c7000 1 -- 172.21.15.27:0/2946005087 --> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] -- osd_op(unknown.0.0:1 7.13 7:ca8fdbba:::100000001fb.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e34) v8 -- 0x55bfec214990 con 0x55bfec2166e0 2023-10-06T01:42:20.197 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.191+0000 7f623e101700 1 --2- 172.21.15.27:0/2946005087 >> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] conn(0x55bfec2166e0 0x55bfec214450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:20.197 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.191+0000 7f623e902700 1 --2- 172.21.15.27:0/2946005087 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6220065890 0x7f6220067d40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f623800c010 tx=0x7f6238016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:20.197 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.191+0000 7f623e101700 1 --2- 172.21.15.27:0/2946005087 >> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] conn(0x55bfec2166e0 0x55bfec214450 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:20.197 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.191+0000 7f623e101700 1 -- 172.21.15.27:0/2946005087 <== osd.0 v2:172.21.15.27:6800/90644 1 ==== osd_op_reply(1 100000001fb.00000000 [getxattr (103) out=103b] v0'0 uv2 ondisk = 0) v8 ==== 164+0+103 (crc 0 0 0) 0x7f622800e6d0 con 0x55bfec2166e0 2023-10-06T01:42:20.197 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.191+0000 7f62456c7000 1 -- 172.21.15.27:0/2946005087 >> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] conn(0x55bfec2166e0 msgr2=0x55bfec214450 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:20.198 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.191+0000 7f62456c7000 1 --2- 172.21.15.27:0/2946005087 >> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] conn(0x55bfec2166e0 0x55bfec214450 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:20.198 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.191+0000 7f62456c7000 1 -- 172.21.15.27:0/2946005087 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6220065890 msgr2=0x7f6220067d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:20.198 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.191+0000 7f62456c7000 1 --2- 172.21.15.27:0/2946005087 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6220065890 0x7f6220067d40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f623800c010 tx=0x7f6238016000 comp rx=0 tx=0).stop 2023-10-06T01:42:20.198 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.191+0000 7f62456c7000 1 -- 172.21.15.27:0/2946005087 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bfec186890 msgr2=0x55bfec17e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:20.198 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.191+0000 7f62456c7000 1 --2- 172.21.15.27:0/2946005087 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55bfec186890 0x55bfec17e960 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f6230003000 tx=0x7f6230018000 comp rx=0 tx=0).stop 2023-10-06T01:42:20.198 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.191+0000 7f623f103700 1 -- 172.21.15.27:0/2946005087 reap_dead start 2023-10-06T01:42:20.198 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.191+0000 7f62456c7000 1 -- 172.21.15.27:0/2946005087 shutdown_connections 2023-10-06T01:42:20.198 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.191+0000 7f62456c7000 1 -- 172.21.15.27:0/2946005087 >> 172.21.15.27:0/2946005087 conn(0x55bfec16c7b0 msgr2=0x55bfec184290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:20.198 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.191+0000 7f62456c7000 1 -- 172.21.15.27:0/2946005087 shutdown_connections 2023-10-06T01:42:20.199 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.191+0000 7f62456c7000 1 -- 172.21.15.27:0/2946005087 wait complete. 2023-10-06T01:42:20.199 INFO:teuthology.orchestra.run.smithi027.stdout:a��alphaparent_c% 2023-10-06T01:42:20.200 DEBUG:teuthology.orchestra.run.smithi027:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2023-10-06T01:42:20.293 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2023-10-06T01:42:20.293 INFO:teuthology.orchestra.run.smithi027.stdout: "ino": 1099511628283, 2023-10-06T01:42:20.293 INFO:teuthology.orchestra.run.smithi027.stdout: "ancestors": [ 2023-10-06T01:42:20.293 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:42:20.293 INFO:teuthology.orchestra.run.smithi027.stdout: "dirino": 1099511628285, 2023-10-06T01:42:20.293 INFO:teuthology.orchestra.run.smithi027.stdout: "dname": "alpha", 2023-10-06T01:42:20.293 INFO:teuthology.orchestra.run.smithi027.stdout: "version": 8 2023-10-06T01:42:20.293 INFO:teuthology.orchestra.run.smithi027.stdout: }, 2023-10-06T01:42:20.294 INFO:teuthology.orchestra.run.smithi027.stdout: { 2023-10-06T01:42:20.294 INFO:teuthology.orchestra.run.smithi027.stdout: "dirino": 1, 2023-10-06T01:42:20.294 INFO:teuthology.orchestra.run.smithi027.stdout: "dname": "parent_c", 2023-10-06T01:42:20.294 INFO:teuthology.orchestra.run.smithi027.stdout: "version": 37 2023-10-06T01:42:20.294 INFO:teuthology.orchestra.run.smithi027.stdout: } 2023-10-06T01:42:20.294 INFO:teuthology.orchestra.run.smithi027.stdout: ], 2023-10-06T01:42:20.294 INFO:teuthology.orchestra.run.smithi027.stdout: "pool": 7, 2023-10-06T01:42:20.294 INFO:teuthology.orchestra.run.smithi027.stdout: "old_pools": [ 2023-10-06T01:42:20.294 INFO:teuthology.orchestra.run.smithi027.stdout: 6 2023-10-06T01:42:20.294 INFO:teuthology.orchestra.run.smithi027.stdout: ] 2023-10-06T01:42:20.294 INFO:teuthology.orchestra.run.smithi027.stdout:} 2023-10-06T01:42:20.294 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:20.295 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T01:42:20.296 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.file.layout.object_size -v 8388608 ./parent_c/alpha) 2023-10-06T01:42:20.355 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T01:42:20.513 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.512+0000 7fbdfee28700 1 Processor -- start 2023-10-06T01:42:20.513 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.512+0000 7fbdfee28700 1 -- start start 2023-10-06T01:42:20.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.513+0000 7fbdfee28700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbdf80cc170 0x7fbdf815c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:20.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.513+0000 7fbdfee28700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fbdf815cc60 0x7fbdf815d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:20.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.513+0000 7fbdfee28700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdf80af040 con 0x7fbdf80cbda0 2023-10-06T01:42:20.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.513+0000 7fbdfee28700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdf80afd70 con 0x7fbdf815cc60 2023-10-06T01:42:20.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.513+0000 7fbdfee28700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf8068080 con 0x7fbdf80cc170 2023-10-06T01:42:20.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.513+0000 7fbdfde26700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fbdf80cbda0 0x7fbdf8067630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40410/0 (socket says 172.21.15.27:40410) 2023-10-06T01:42:20.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.513+0000 7fbdfde26700 1 -- 172.21.15.27:0/1183924458 learned_addr learned my addr 172.21.15.27:0/1183924458 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:20.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.513+0000 7fbdfd625700 1 --2- 172.21.15.27:0/1183924458 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbdf80cc170 0x7fbdf815c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:20.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.513+0000 7fbdfe627700 1 --2- 172.21.15.27:0/1183924458 >> v2:172.21.15.124:3300/0 conn(0x7fbdf815cc60 0x7fbdf815d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:20.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.513+0000 7fbdfce24700 1 -- 172.21.15.27:0/1183924458 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2069606261 0 0) 0x7fbdf80af040 con 0x7fbdf80cbda0 2023-10-06T01:42:20.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.513+0000 7fbdfce24700 1 -- 172.21.15.27:0/1183924458 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbde4003570 con 0x7fbdf80cbda0 2023-10-06T01:42:20.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.513+0000 7fbdfce24700 1 -- 172.21.15.27:0/1183924458 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 876796499 0 0) 0x7fbde4003570 con 0x7fbdf80cbda0 2023-10-06T01:42:20.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.513+0000 7fbdfd625700 1 -- 172.21.15.27:0/1183924458 >> v2:172.21.15.124:3300/0 conn(0x7fbdf815cc60 msgr2=0x7fbdf815d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:20.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.513+0000 7fbdfd625700 1 --2- 172.21.15.27:0/1183924458 >> v2:172.21.15.124:3300/0 conn(0x7fbdf815cc60 0x7fbdf815d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:20.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.513+0000 7fbdfd625700 1 -- 172.21.15.27:0/1183924458 >> v1:172.21.15.27:6789/0 conn(0x7fbdf80cbda0 legacy=0x7fbdf8067630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:20.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.513+0000 7fbdfd625700 1 -- 172.21.15.27:0/1183924458 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf80af290 con 0x7fbdf80cc170 2023-10-06T01:42:20.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.513+0000 7fbdfe627700 1 --2- 172.21.15.27:0/1183924458 >> v2:172.21.15.124:3300/0 conn(0x7fbdf815cc60 0x7fbdf815d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:42:20.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.514+0000 7fbdfd625700 1 --2- 172.21.15.27:0/1183924458 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbdf80cc170 0x7fbdf815c720 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbde80098e0 tx=0x7fbde80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ade69bd1466a4d8 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:20.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.514+0000 7fbdfce24700 1 -- 172.21.15.27:0/1183924458 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf4001470 con 0x7fbdf80cc170 2023-10-06T01:42:20.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.514+0000 7fbdfce24700 1 -- 172.21.15.27:0/1183924458 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde8011420 con 0x7fbdf80cc170 2023-10-06T01:42:20.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.514+0000 7fbdfce24700 1 -- 172.21.15.27:0/1183924458 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde80117a0 con 0x7fbdf80cc170 2023-10-06T01:42:20.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.514+0000 7fbdfee28700 1 -- 172.21.15.27:0/1183924458 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbdf80cc170 msgr2=0x7fbdf815c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:20.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.514+0000 7fbdfee28700 1 --2- 172.21.15.27:0/1183924458 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbdf80cc170 0x7fbdf815c720 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fbde80098e0 tx=0x7fbde80091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:20.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.514+0000 7fbdfee28700 1 -- 172.21.15.27:0/1183924458 shutdown_connections 2023-10-06T01:42:20.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.514+0000 7fbdfee28700 1 --2- 172.21.15.27:0/1183924458 >> v2:172.21.15.124:3300/0 conn(0x7fbdf815cc60 0x7fbdf815d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:20.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.514+0000 7fbdfee28700 1 --2- 172.21.15.27:0/1183924458 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbdf80cc170 0x7fbdf815c720 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:20.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.514+0000 7fbdfee28700 1 -- 172.21.15.27:0/1183924458 >> 172.21.15.27:0/1183924458 conn(0x7fbdf80d0f70 msgr2=0x7fbdf80ccf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:20.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.514+0000 7fbdfee28700 1 -- 172.21.15.27:0/1183924458 shutdown_connections 2023-10-06T01:42:20.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.514+0000 7fbdfee28700 1 -- 172.21.15.27:0/1183924458 wait complete. 2023-10-06T01:42:20.521 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.516+0000 7fbdfee28700 1 Processor -- start 2023-10-06T01:42:20.521 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.516+0000 7fbdfee28700 1 -- start start 2023-10-06T01:42:20.521 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.516+0000 7fbdfee28700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbdf80cbda0 0x7fbdf80b4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:20.521 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.516+0000 7fbdfee28700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdf80cc170 0x7fbdf8157450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:20.521 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.516+0000 7fbdfee28700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbdf815cc60 0x7fbdf8155300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:20.521 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.516+0000 7fbdfee28700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf8158c60 con 0x7fbdf80cc170 2023-10-06T01:42:20.521 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.516+0000 7fbdfee28700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf8157990 con 0x7fbdf80cbda0 2023-10-06T01:42:20.521 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.516+0000 7fbdfee28700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf8157bc0 con 0x7fbdf815cc60 2023-10-06T01:42:20.521 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.517+0000 7fbdfd625700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdf80cc170 0x7fbdf8157450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:20.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.517+0000 7fbdfd625700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdf80cc170 0x7fbdf8157450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59976/0 (socket says 172.21.15.27:59976) 2023-10-06T01:42:20.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.517+0000 7fbdfd625700 1 -- 172.21.15.27:0/2116596103 learned_addr learned my addr 172.21.15.27:0/2116596103 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:20.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.517+0000 7fbdfe627700 1 --2- 172.21.15.27:0/2116596103 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbdf815cc60 0x7fbdf8155300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:20.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.517+0000 7fbdfde26700 1 --2- 172.21.15.27:0/2116596103 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbdf80cbda0 0x7fbdf80b4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:20.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.517+0000 7fbdfd625700 1 -- 172.21.15.27:0/2116596103 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbdf815cc60 msgr2=0x7fbdf8155300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:20.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.517+0000 7fbdfd625700 1 --2- 172.21.15.27:0/2116596103 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbdf815cc60 0x7fbdf8155300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:20.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.517+0000 7fbdfd625700 1 -- 172.21.15.27:0/2116596103 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbdf80cbda0 msgr2=0x7fbdf80b4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:20.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.517+0000 7fbdfd625700 1 --2- 172.21.15.27:0/2116596103 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbdf80cbda0 0x7fbdf80b4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:20.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.517+0000 7fbdfd625700 1 -- 172.21.15.27:0/2116596103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf0009a20 con 0x7fbdf80cc170 2023-10-06T01:42:20.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.517+0000 7fbdfd625700 1 --2- 172.21.15.27:0/2116596103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdf80cc170 0x7fbdf8157450 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fbdf80681c0 tx=0x7fbde8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:20.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.517+0000 7fbdedffb700 1 -- 172.21.15.27:0/2116596103 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde801caf0 con 0x7fbdf80cc170 2023-10-06T01:42:20.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.517+0000 7fbdedffb700 1 -- 172.21.15.27:0/2116596103 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde801cc50 con 0x7fbdf80cc170 2023-10-06T01:42:20.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.517+0000 7fbdfee28700 1 -- 172.21.15.27:0/2116596103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbde8009420 con 0x7fbdf80cc170 2023-10-06T01:42:20.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.517+0000 7fbdedffb700 1 -- 172.21.15.27:0/2116596103 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde801a450 con 0x7fbdf80cc170 2023-10-06T01:42:20.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.517+0000 7fbdfee28700 1 -- 172.21.15.27:0/2116596103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdf8158180 con 0x7fbdf80cc170 2023-10-06T01:42:20.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.518+0000 7fbdfee28700 1 -- 172.21.15.27:0/2116596103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdf804c3c0 con 0x7fbdf80cc170 2023-10-06T01:42:20.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.519+0000 7fbdedffb700 1 -- 172.21.15.27:0/2116596103 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fbde801a7e0 con 0x7fbdf80cc170 2023-10-06T01:42:20.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.519+0000 7fbdedffb700 1 --2- 172.21.15.27:0/2116596103 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fbddc066f30 0x7fbddc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:20.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.519+0000 7fbdedffb700 1 -- 172.21.15.27:0/2116596103 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6453+0+0 (secure 0 0 0) 0x7fbde8085fb0 con 0x7fbdf80cc170 2023-10-06T01:42:20.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.519+0000 7fbdfde26700 1 --2- 172.21.15.27:0/2116596103 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fbddc066f30 0x7fbddc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:20.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.520+0000 7fbdfde26700 1 --2- 172.21.15.27:0/2116596103 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fbddc066f30 0x7fbddc0693e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbdf4001ed0 tx=0x7fbdf40019d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:20.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.524+0000 7fbdedffb700 1 -- 172.21.15.27:0/2116596103 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbde80b75a0 con 0x7fbdf80cc170 2023-10-06T01:42:20.763 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.762+0000 7fbdfee28700 1 -- 172.21.15.27:0/2116596103 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbdf80cdc00 con 0x7fbdf80cc170 2023-10-06T01:42:20.765 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.762+0000 7fbdedffb700 1 -- 172.21.15.27:0/2116596103 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4599 (secure 0 0 0) 0x7fbde8052ff0 con 0x7fbdf80cc170 2023-10-06T01:42:20.765 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:20.766 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4870,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6836/1043044736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":1043044736},{"type":"v1","addr":"172.21.15.124:6836","nonce":1043044736}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4962,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/1207987117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1207987117},{"type":"v1","addr":"172.21.15.27:6834","nonce":1207987117}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4965,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/943261840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":943261840},{"type":"v1","addr":"172.21.15.27:6835","nonce":943261840}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T01:42:03.036784+0000","modified":"2023-10-06T01:42:17.363571+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4974},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4974":{"gid":4974,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.124:6837/325247112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6835","nonce":325247112},{"type":"v1","addr":"172.21.15.124:6837","nonce":325247112}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6,7],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-10-06T01:42:20.766 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.764+0000 7fbdfee28700 1 -- 172.21.15.27:0/2116596103 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fbddc066f30 msgr2=0x7fbddc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:20.766 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.764+0000 7fbdfee28700 1 --2- 172.21.15.27:0/2116596103 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fbddc066f30 0x7fbddc0693e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbdf4001ed0 tx=0x7fbdf40019d0 comp rx=0 tx=0).stop 2023-10-06T01:42:20.766 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.764+0000 7fbdfee28700 1 -- 172.21.15.27:0/2116596103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdf80cc170 msgr2=0x7fbdf8157450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:20.767 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.764+0000 7fbdfee28700 1 --2- 172.21.15.27:0/2116596103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdf80cc170 0x7fbdf8157450 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fbdf80681c0 tx=0x7fbde8016000 comp rx=0 tx=0).stop 2023-10-06T01:42:20.767 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.764+0000 7fbdfee28700 1 -- 172.21.15.27:0/2116596103 shutdown_connections 2023-10-06T01:42:20.767 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.765+0000 7fbdfee28700 1 --2- 172.21.15.27:0/2116596103 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fbddc066f30 0x7fbddc0693e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:20.767 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.765+0000 7fbdfee28700 1 --2- 172.21.15.27:0/2116596103 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbdf815cc60 0x7fbdf8155300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:20.767 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.765+0000 7fbdfee28700 1 --2- 172.21.15.27:0/2116596103 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbdf80cbda0 0x7fbdf80b4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:20.767 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.765+0000 7fbdfee28700 1 --2- 172.21.15.27:0/2116596103 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbdf80cc170 0x7fbdf8157450 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:20.767 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.765+0000 7fbdfee28700 1 -- 172.21.15.27:0/2116596103 >> 172.21.15.27:0/2116596103 conn(0x7fbdf80d0f70 msgr2=0x7fbdf80c6d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:20.767 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.765+0000 7fbdfee28700 1 -- 172.21.15.27:0/2116596103 shutdown_connections 2023-10-06T01:42:20.767 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.765+0000 7fbdfee28700 1 -- 172.21.15.27:0/2116596103 wait complete. 2023-10-06T01:42:20.768 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 16 2023-10-06T01:42:20.778 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json flush journal 2023-10-06T01:42:20.922 INFO:teuthology.orchestra.run.smithi124.stdout:{"message":"","return_code":0} 2023-10-06T01:42:20.922 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2023-10-06T01:42:20.922 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool data_new getxattr 100000001fb.00000000 layout 2023-10-06T01:42:20.961 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.960+0000 7fd8b54df000 1 Processor -- start 2023-10-06T01:42:20.961 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.960+0000 7fd8b54df000 1 -- start start 2023-10-06T01:42:20.964 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.960+0000 7fd8b54df000 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55cd63ccb690 0x55cd63d97200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:20.964 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.960+0000 7fd8b54df000 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55cd63d112d0 0x55cd63d0e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:20.964 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.960+0000 7fd8b54df000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cd63d12890 0x55cd63d9f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:20.964 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.960+0000 7fd8b54df000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd63c6f320 con 0x55cd63d12890 2023-10-06T01:42:20.964 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.960+0000 7fd8b54df000 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd63c13980 con 0x55cd63ccb690 2023-10-06T01:42:20.964 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.960+0000 7fd8b54df000 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cd63c13620 con 0x55cd63d112d0 2023-10-06T01:42:20.964 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.961+0000 7fd8ae71a700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55cd63ccb690 0x55cd63d97200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:20.965 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.961+0000 7fd8aef1b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cd63d12890 0x55cd63d9f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:20.965 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.961+0000 7fd8ae71a700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55cd63ccb690 0x55cd63d97200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.27:58752/0 (socket says 172.21.15.27:58752) 2023-10-06T01:42:20.965 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.961+0000 7fd8aef1b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cd63d12890 0x55cd63d9f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:59986/0 (socket says 172.21.15.27:59986) 2023-10-06T01:42:20.965 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.961+0000 7fd8ae71a700 1 -- 172.21.15.27:0/3763210235 learned_addr learned my addr 172.21.15.27:0/3763210235 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:20.965 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.961+0000 7fd8adf19700 1 --2- 172.21.15.27:0/3763210235 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55cd63d112d0 0x55cd63d0e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:20.965 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.961+0000 7fd8aef1b700 1 -- 172.21.15.27:0/3763210235 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55cd63d112d0 msgr2=0x55cd63d0e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:20.965 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.961+0000 7fd8aef1b700 1 --2- 172.21.15.27:0/3763210235 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55cd63d112d0 0x55cd63d0e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:20.965 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.961+0000 7fd8aef1b700 1 -- 172.21.15.27:0/3763210235 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55cd63ccb690 msgr2=0x55cd63d97200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:20.965 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.961+0000 7fd8aef1b700 1 --2- 172.21.15.27:0/3763210235 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55cd63ccb690 0x55cd63d97200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:20.965 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.961+0000 7fd8aef1b700 1 -- 172.21.15.27:0/3763210235 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a8005cf0 con 0x55cd63d12890 2023-10-06T01:42:20.965 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.961+0000 7fd8adf19700 1 --2- 172.21.15.27:0/3763210235 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55cd63d112d0 0x55cd63d0e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:42:20.966 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.961+0000 7fd8aef1b700 1 --2- 172.21.15.27:0/3763210235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cd63d12890 0x55cd63d9f740 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0006eb0 tx=0x7fd8a0006930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11119cc472503176 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:20.966 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.961+0000 7fd8acf17700 1 -- 172.21.15.27:0/3763210235 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a0006110 con 0x55cd63d12890 2023-10-06T01:42:20.966 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.961+0000 7fd8acf17700 1 -- 172.21.15.27:0/3763210235 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8a0006290 con 0x55cd63d12890 2023-10-06T01:42:20.966 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.961+0000 7fd8acf17700 1 -- 172.21.15.27:0/3763210235 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a0006400 con 0x55cd63d12890 2023-10-06T01:42:20.966 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.962+0000 7fd8b54df000 1 -- 172.21.15.27:0/3763210235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cd63d12890 msgr2=0x55cd63d9f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:20.966 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.962+0000 7fd8b54df000 1 --2- 172.21.15.27:0/3763210235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cd63d12890 0x55cd63d9f740 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0006eb0 tx=0x7fd8a0006930 comp rx=0 tx=0).stop 2023-10-06T01:42:20.966 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.962+0000 7fd8b54df000 1 -- 172.21.15.27:0/3763210235 shutdown_connections 2023-10-06T01:42:20.966 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.962+0000 7fd8b54df000 1 --2- 172.21.15.27:0/3763210235 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cd63d12890 0x55cd63d9f740 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:20.966 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.962+0000 7fd8b54df000 1 --2- 172.21.15.27:0/3763210235 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55cd63ccb690 0x55cd63d97200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:20.966 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.962+0000 7fd8b54df000 1 --2- 172.21.15.27:0/3763210235 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55cd63d112d0 0x55cd63d0e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:20.967 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.962+0000 7fd8b54df000 1 -- 172.21.15.27:0/3763210235 >> 172.21.15.27:0/3763210235 conn(0x55cd63cf87b0 msgr2=0x55cd63d0ff60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:20.967 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.962+0000 7fd8b54df000 1 -- 172.21.15.27:0/3763210235 shutdown_connections 2023-10-06T01:42:20.967 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.962+0000 7fd8b54df000 1 -- 172.21.15.27:0/3763210235 wait complete. 2023-10-06T01:42:20.967 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.962+0000 7fd8b54df000 1 Processor -- start 2023-10-06T01:42:20.967 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.962+0000 7fd8b54df000 1 -- start start 2023-10-06T01:42:20.970 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.963+0000 7fd8b54df000 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55cd63ccb690 0x55cd63d977d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:20.970 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.963+0000 7fd8b54df000 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55cd63d112d0 0x55cd63d92bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:20.970 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.963+0000 7fd8b54df000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cd63d12890 0x55cd63d9f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:20.970 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.963+0000 7fd8b54df000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd63c0a400 con 0x55cd63d12890 2023-10-06T01:42:20.970 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.963+0000 7fd8b54df000 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd63d97e70 con 0x55cd63d112d0 2023-10-06T01:42:20.971 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.963+0000 7fd8b54df000 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x55cd63d99d00 con 0x55cd63ccb690 2023-10-06T01:42:20.971 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.963+0000 7fd8aef1b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cd63d12890 0x55cd63d9f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:20.973 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.963+0000 7fd8aef1b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cd63d12890 0x55cd63d9f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60000/0 (socket says 172.21.15.27:60000) 2023-10-06T01:42:20.973 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.963+0000 7fd8aef1b700 1 -- 172.21.15.27:0/1703304928 learned_addr learned my addr 172.21.15.27:0/1703304928 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:20.973 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.963+0000 7fd8ae71a700 1 --2- 172.21.15.27:0/1703304928 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55cd63ccb690 0x55cd63d977d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:20.973 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.963+0000 7fd8adf19700 1 --2- 172.21.15.27:0/1703304928 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55cd63d112d0 0x55cd63d92bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:20.973 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.963+0000 7fd8aef1b700 1 -- 172.21.15.27:0/1703304928 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55cd63ccb690 msgr2=0x55cd63d977d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:20.973 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.963+0000 7fd8aef1b700 1 --2- 172.21.15.27:0/1703304928 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55cd63ccb690 0x55cd63d977d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:20.973 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.963+0000 7fd8aef1b700 1 -- 172.21.15.27:0/1703304928 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55cd63d112d0 msgr2=0x55cd63d92bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:20.973 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.963+0000 7fd8aef1b700 1 --2- 172.21.15.27:0/1703304928 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55cd63d112d0 0x55cd63d92bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:20.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.964+0000 7fd8aef1b700 1 -- 172.21.15.27:0/1703304928 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd89c001e10 con 0x55cd63d12890 2023-10-06T01:42:20.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.964+0000 7fd8aef1b700 1 --2- 172.21.15.27:0/1703304928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cd63d12890 0x55cd63d9f740 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0001c20 tx=0x7fd8a0010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:20.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.964+0000 7fd8a6ffd700 1 -- 172.21.15.27:0/1703304928 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a0004030 con 0x55cd63d12890 2023-10-06T01:42:20.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.964+0000 7fd8a6ffd700 1 -- 172.21.15.27:0/1703304928 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8a0004190 con 0x55cd63d12890 2023-10-06T01:42:20.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.964+0000 7fd8b54df000 1 -- 172.21.15.27:0/1703304928 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55cd63c15ae0 con 0x55cd63d12890 2023-10-06T01:42:20.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.964+0000 7fd8a6ffd700 1 -- 172.21.15.27:0/1703304928 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a0004550 con 0x55cd63d12890 2023-10-06T01:42:20.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.964+0000 7fd8b54df000 1 -- 172.21.15.27:0/1703304928 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cd63c15540 con 0x55cd63d12890 2023-10-06T01:42:20.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.965+0000 7fd8a6ffd700 1 -- 172.21.15.27:0/1703304928 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd8a0003450 con 0x55cd63d12890 2023-10-06T01:42:20.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.966+0000 7fd8a6ffd700 1 --2- 172.21.15.27:0/1703304928 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd890065890 0x7fd890067d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:20.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.966+0000 7fd8a6ffd700 1 -- 172.21.15.27:0/1703304928 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6453+0+0 (secure 0 0 0) 0x7fd8a0084170 con 0x55cd63d12890 2023-10-06T01:42:20.974 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.966+0000 7fd8ae71a700 1 --2- 172.21.15.27:0/1703304928 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd890065890 0x7fd890067d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:20.975 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.966+0000 7fd8b54df000 1 --2- 172.21.15.27:0/1703304928 >> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] conn(0x55cd63d93100 0x55cd63d9ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:20.975 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.966+0000 7fd8b54df000 1 -- 172.21.15.27:0/1703304928 --> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] -- osd_op(unknown.0.0:1 7.13 7:ca8fdbba:::100000001fb.00000000:head [getxattr layout in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e34) v8 -- 0x55cd63d98180 con 0x55cd63d93100 2023-10-06T01:42:20.975 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.966+0000 7fd8adf19700 1 --2- 172.21.15.27:0/1703304928 >> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] conn(0x55cd63d93100 0x55cd63d9ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:20.975 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.966+0000 7fd8ae71a700 1 --2- 172.21.15.27:0/1703304928 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd890065890 0x7fd890067d40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd8a800a650 tx=0x7fd8a8016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:20.975 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.966+0000 7fd8adf19700 1 --2- 172.21.15.27:0/1703304928 >> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] conn(0x55cd63d93100 0x55cd63d9ae90 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:20.975 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.967+0000 7fd8adf19700 1 -- 172.21.15.27:0/1703304928 <== osd.0 v2:172.21.15.27:6800/90644 1 ==== osd_op_reply(1 100000001fb.00000000 [getxattr (30) out=30b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+30 (crc 0 0 0) 0x7fd89800e7f0 con 0x55cd63d93100 2023-10-06T01:42:20.975 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.967+0000 7fd8b54df000 1 -- 172.21.15.27:0/1703304928 >> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] conn(0x55cd63d93100 msgr2=0x55cd63d9ae90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:20.975 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.967+0000 7fd8b54df000 1 --2- 172.21.15.27:0/1703304928 >> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] conn(0x55cd63d93100 0x55cd63d9ae90 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:20.975 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.967+0000 7fd8b54df000 1 -- 172.21.15.27:0/1703304928 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd890065890 msgr2=0x7fd890067d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:20.975 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.967+0000 7fd8b54df000 1 --2- 172.21.15.27:0/1703304928 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd890065890 0x7fd890067d40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd8a800a650 tx=0x7fd8a8016000 comp rx=0 tx=0).stop 2023-10-06T01:42:20.975 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.967+0000 7fd8b54df000 1 -- 172.21.15.27:0/1703304928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cd63d12890 msgr2=0x55cd63d9f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:20.976 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.967+0000 7fd8b54df000 1 --2- 172.21.15.27:0/1703304928 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55cd63d12890 0x55cd63d9f740 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fd8a0001c20 tx=0x7fd8a0010000 comp rx=0 tx=0).stop 2023-10-06T01:42:20.976 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.967+0000 7fd8aef1b700 1 -- 172.21.15.27:0/1703304928 reap_dead start 2023-10-06T01:42:20.976 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.967+0000 7fd8b54df000 1 -- 172.21.15.27:0/1703304928 shutdown_connections 2023-10-06T01:42:20.976 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.967+0000 7fd8b54df000 1 -- 172.21.15.27:0/1703304928 >> 172.21.15.27:0/1703304928 conn(0x55cd63cf87b0 msgr2=0x55cd63d0d400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:20.976 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.967+0000 7fd8b54df000 1 -- 172.21.15.27:0/1703304928 shutdown_connections 2023-10-06T01:42:20.976 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:20.967+0000 7fd8b54df000 1 -- 172.21.15.27:0/1703304928 wait complete. 2023-10-06T01:42:20.977 INFO:teuthology.orchestra.run.smithi027.stdout:@� 2023-10-06T01:42:20.977 DEBUG:teuthology.orchestra.run.smithi027:> ceph-dencoder type file_layout_t import - decode dump_json 2023-10-06T01:42:21.069 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2023-10-06T01:42:21.070 INFO:teuthology.orchestra.run.smithi027.stdout: "stripe_unit": 4194304, 2023-10-06T01:42:21.070 INFO:teuthology.orchestra.run.smithi027.stdout: "stripe_count": 1, 2023-10-06T01:42:21.070 INFO:teuthology.orchestra.run.smithi027.stdout: "object_size": 8388608, 2023-10-06T01:42:21.070 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_id": 7, 2023-10-06T01:42:21.070 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_ns": "" 2023-10-06T01:42:21.070 INFO:teuthology.orchestra.run.smithi027.stdout:} 2023-10-06T01:42:21.070 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:21.072 DEBUG:teuthology.orchestra.run.smithi027:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001fb.00000000 layout 2023-10-06T01:42:21.150 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.149+0000 7f9c2309d000 1 Processor -- start 2023-10-06T01:42:21.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.149+0000 7f9c2309d000 1 -- start start 2023-10-06T01:42:21.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.150+0000 7f9c2309d000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0733bae00 0x55b073446520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:21.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.150+0000 7f9c2309d000 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55b073430250 0x55b073375f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:21.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.150+0000 7f9c2309d000 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55b073430620 0x55b073376480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:21.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.150+0000 7f9c2309d000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0732be980 con 0x55b0733bae00 2023-10-06T01:42:21.155 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.150+0000 7f9c2309d000 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0732be620 con 0x55b073430250 2023-10-06T01:42:21.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.150+0000 7f9c2309d000 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b0732b5400 con 0x55b073430620 2023-10-06T01:42:21.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.150+0000 7f9c1c2d8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0733bae00 0x55b073446520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:21.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.150+0000 7f9c1c2d8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0733bae00 0x55b073446520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60018/0 (socket says 172.21.15.27:60018) 2023-10-06T01:42:21.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.150+0000 7f9c1c2d8700 1 -- 172.21.15.27:0/2303217369 learned_addr learned my addr 172.21.15.27:0/2303217369 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:21.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.150+0000 7f9c1bad7700 1 --2- 172.21.15.27:0/2303217369 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55b073430250 0x55b073375f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:21.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.150+0000 7f9c1cad9700 1 --2- 172.21.15.27:0/2303217369 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55b073430620 0x55b073376480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:21.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.150+0000 7f9c1c2d8700 1 -- 172.21.15.27:0/2303217369 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55b073430620 msgr2=0x55b073376480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:21.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.150+0000 7f9c1c2d8700 1 --2- 172.21.15.27:0/2303217369 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55b073430620 0x55b073376480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:21.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.150+0000 7f9c1c2d8700 1 -- 172.21.15.27:0/2303217369 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55b073430250 msgr2=0x55b073375f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:21.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.150+0000 7f9c1c2d8700 1 --2- 172.21.15.27:0/2303217369 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55b073430250 0x55b073375f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:21.156 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.150+0000 7f9c1c2d8700 1 -- 172.21.15.27:0/2303217369 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c04005c90 con 0x55b0733bae00 2023-10-06T01:42:21.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.150+0000 7f9c1c2d8700 1 --2- 172.21.15.27:0/2303217369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0733bae00 0x55b073446520 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f9c1400d8e0 tx=0x7f9c14007040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57ed9d988255f85f server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:21.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.150+0000 7f9c1aad5700 1 -- 172.21.15.27:0/2303217369 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1400b8c0 con 0x55b0733bae00 2023-10-06T01:42:21.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.151+0000 7f9c1aad5700 1 -- 172.21.15.27:0/2303217369 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c1400ba20 con 0x55b0733bae00 2023-10-06T01:42:21.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.151+0000 7f9c1aad5700 1 -- 172.21.15.27:0/2303217369 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1400bdc0 con 0x55b0733bae00 2023-10-06T01:42:21.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.151+0000 7f9c2309d000 1 -- 172.21.15.27:0/2303217369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0733bae00 msgr2=0x55b073446520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:21.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.151+0000 7f9c2309d000 1 --2- 172.21.15.27:0/2303217369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0733bae00 0x55b073446520 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f9c1400d8e0 tx=0x7f9c14007040 comp rx=0 tx=0).stop 2023-10-06T01:42:21.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.151+0000 7f9c2309d000 1 -- 172.21.15.27:0/2303217369 shutdown_connections 2023-10-06T01:42:21.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.151+0000 7f9c2309d000 1 --2- 172.21.15.27:0/2303217369 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55b073430620 0x55b073376480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:21.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.151+0000 7f9c2309d000 1 --2- 172.21.15.27:0/2303217369 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b0733bae00 0x55b073446520 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:21.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.151+0000 7f9c2309d000 1 --2- 172.21.15.27:0/2303217369 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55b073430250 0x55b073375f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:21.157 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.151+0000 7f9c2309d000 1 -- 172.21.15.27:0/2303217369 >> 172.21.15.27:0/2303217369 conn(0x55b0733748a0 msgr2=0x55b07339f9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:21.158 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.151+0000 7f9c2309d000 1 -- 172.21.15.27:0/2303217369 shutdown_connections 2023-10-06T01:42:21.158 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.151+0000 7f9c2309d000 1 -- 172.21.15.27:0/2303217369 wait complete. 2023-10-06T01:42:21.158 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.152+0000 7f9c2309d000 1 Processor -- start 2023-10-06T01:42:21.158 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.152+0000 7f9c2309d000 1 -- start start 2023-10-06T01:42:21.158 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.152+0000 7f9c2309d000 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55b0733bae00 0x55b07345ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:21.158 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.152+0000 7f9c2309d000 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55b073430250 0x55b0734293f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:21.158 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.152+0000 7f9c2309d000 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b073430620 0x55b073446520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:21.158 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.152+0000 7f9c2309d000 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0733b7800 con 0x55b073430620 2023-10-06T01:42:21.158 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.152+0000 7f9c2309d000 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b07345b070 con 0x55b073430250 2023-10-06T01:42:21.158 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.152+0000 7f9c2309d000 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b0733b72c0 con 0x55b0733bae00 2023-10-06T01:42:21.158 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.153+0000 7f9c1cad9700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b073430620 0x55b073446520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:21.159 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.153+0000 7f9c1cad9700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b073430620 0x55b073446520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60030/0 (socket says 172.21.15.27:60030) 2023-10-06T01:42:21.159 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.153+0000 7f9c1cad9700 1 -- 172.21.15.27:0/3601826345 learned_addr learned my addr 172.21.15.27:0/3601826345 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:21.159 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.153+0000 7f9c1c2d8700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55b0733bae00 0x55b07345ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:21.159 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.153+0000 7f9c1bad7700 1 --2- 172.21.15.27:0/3601826345 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55b073430250 0x55b0734293f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:21.159 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.153+0000 7f9c1cad9700 1 -- 172.21.15.27:0/3601826345 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55b0733bae00 msgr2=0x55b07345ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:21.159 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.153+0000 7f9c1cad9700 1 --2- 172.21.15.27:0/3601826345 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x55b0733bae00 0x55b07345ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:21.159 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.153+0000 7f9c1cad9700 1 -- 172.21.15.27:0/3601826345 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55b073430250 msgr2=0x55b0734293f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:21.159 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.153+0000 7f9c1cad9700 1 --2- 172.21.15.27:0/3601826345 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x55b073430250 0x55b0734293f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:21.159 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.153+0000 7f9c1cad9700 1 -- 172.21.15.27:0/3601826345 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c1400d460 con 0x55b073430620 2023-10-06T01:42:21.159 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.153+0000 7f9c1cad9700 1 --2- 172.21.15.27:0/3601826345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b073430620 0x55b073446520 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c00c6e0 tx=0x7f9c0c001690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:21.159 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.153+0000 7f9c192d2700 1 -- 172.21.15.27:0/3601826345 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c001b90 con 0x55b073430620 2023-10-06T01:42:21.160 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.153+0000 7f9c192d2700 1 -- 172.21.15.27:0/3601826345 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c0c013070 con 0x55b073430620 2023-10-06T01:42:21.160 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.153+0000 7f9c2309d000 1 -- 172.21.15.27:0/3601826345 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55b0732cd3d0 con 0x55b073430620 2023-10-06T01:42:21.160 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.153+0000 7f9c192d2700 1 -- 172.21.15.27:0/3601826345 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c00f450 con 0x55b073430620 2023-10-06T01:42:21.160 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.153+0000 7f9c2309d000 1 -- 172.21.15.27:0/3601826345 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b0732c0ae0 con 0x55b073430620 2023-10-06T01:42:21.162 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.155+0000 7f9c192d2700 1 -- 172.21.15.27:0/3601826345 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9c0c00f840 con 0x55b073430620 2023-10-06T01:42:21.162 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.155+0000 7f9c192d2700 1 --2- 172.21.15.27:0/3601826345 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9bfc065890 0x7f9bfc067d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:21.162 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.155+0000 7f9c192d2700 1 -- 172.21.15.27:0/3601826345 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f9c0c083a90 con 0x55b073430620 2023-10-06T01:42:21.162 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.155+0000 7f9c1c2d8700 1 --2- 172.21.15.27:0/3601826345 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9bfc065890 0x7f9bfc067d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:21.162 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.155+0000 7f9c2309d000 1 --2- 172.21.15.27:0/3601826345 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x55b073446a60 0x55b0733b8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:21.162 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.155+0000 7f9c2309d000 1 -- 172.21.15.27:0/3601826345 --> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] -- osd_op(unknown.0.0:1 6.13 6:ca8fdbba:::100000001fb.00000000:head [getxattr layout in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e34) v8 -- 0x55b0733b8fa0 con 0x55b073446a60 2023-10-06T01:42:21.162 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.155+0000 7f9c1bad7700 1 --2- 172.21.15.27:0/3601826345 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x55b073446a60 0x55b0733b8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:21.162 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.156+0000 7f9c1c2d8700 1 --2- 172.21.15.27:0/3601826345 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9bfc065890 0x7f9bfc067d40 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9c1400d430 tx=0x7f9c1401a000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:21.163 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.156+0000 7f9c1bad7700 1 --2- 172.21.15.27:0/3601826345 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x55b073446a60 0x55b0733b8a60 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:21.163 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.156+0000 7f9c1bad7700 1 -- 172.21.15.27:0/3601826345 <== osd.3 v2:172.21.15.27:6824/90668 1 ==== osd_op_reply(1 100000001fb.00000000 [getxattr (30) out=30b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+30 (crc 0 0 0) 0x7f9c0400cb10 con 0x55b073446a60 2023-10-06T01:42:21.163 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.156+0000 7f9c2309d000 1 -- 172.21.15.27:0/3601826345 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x55b073446a60 msgr2=0x55b0733b8a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:21.163 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.156+0000 7f9c2309d000 1 --2- 172.21.15.27:0/3601826345 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x55b073446a60 0x55b0733b8a60 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:21.163 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.156+0000 7f9c2309d000 1 -- 172.21.15.27:0/3601826345 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9bfc065890 msgr2=0x7f9bfc067d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:21.163 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.156+0000 7f9c2309d000 1 --2- 172.21.15.27:0/3601826345 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9bfc065890 0x7f9bfc067d40 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9c1400d430 tx=0x7f9c1401a000 comp rx=0 tx=0).stop 2023-10-06T01:42:21.163 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.156+0000 7f9c2309d000 1 -- 172.21.15.27:0/3601826345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b073430620 msgr2=0x55b073446520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:21.163 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.156+0000 7f9c2309d000 1 --2- 172.21.15.27:0/3601826345 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x55b073430620 0x55b073446520 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c00c6e0 tx=0x7f9c0c001690 comp rx=0 tx=0).stop 2023-10-06T01:42:21.163 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.156+0000 7f9c1cad9700 1 -- 172.21.15.27:0/3601826345 reap_dead start 2023-10-06T01:42:21.163 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.156+0000 7f9c2309d000 1 -- 172.21.15.27:0/3601826345 shutdown_connections 2023-10-06T01:42:21.164 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.156+0000 7f9c2309d000 1 -- 172.21.15.27:0/3601826345 >> 172.21.15.27:0/3601826345 conn(0x55b0733748a0 msgr2=0x55b0733bcb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:21.164 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.156+0000 7f9c2309d000 1 -- 172.21.15.27:0/3601826345 shutdown_connections 2023-10-06T01:42:21.164 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.156+0000 7f9c2309d000 1 -- 172.21.15.27:0/3601826345 wait complete. 2023-10-06T01:42:21.164 INFO:teuthology.orchestra.run.smithi027.stdout:@@ 2023-10-06T01:42:21.165 DEBUG:teuthology.orchestra.run.smithi027:> ceph-dencoder type file_layout_t import - decode dump_json 2023-10-06T01:42:21.258 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2023-10-06T01:42:21.258 INFO:teuthology.orchestra.run.smithi027.stdout: "stripe_unit": 4194304, 2023-10-06T01:42:21.258 INFO:teuthology.orchestra.run.smithi027.stdout: "stripe_count": 1, 2023-10-06T01:42:21.258 INFO:teuthology.orchestra.run.smithi027.stdout: "object_size": 4194304, 2023-10-06T01:42:21.258 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_id": 6, 2023-10-06T01:42:21.259 INFO:teuthology.orchestra.run.smithi027.stdout: "pool_ns": "" 2023-10-06T01:42:21.259 INFO:teuthology.orchestra.run.smithi027.stdout:} 2023-10-06T01:42:21.259 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:21.260 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-06T01:42:21.260 DEBUG:teuthology.orchestra.run.smithi027:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-06T01:42:21.338 DEBUG:teuthology.orchestra.run.smithi124:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-06T01:42:21.378 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-06T01:42:21.381 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-06T01:42:21.381 INFO:teuthology.orchestra.run:Running command with timeout 900 2023-10-06T01:42:21.381 DEBUG:teuthology.orchestra.run.smithi027:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-06T01:42:21.441 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi027.front.sepia.ceph.com 2023-10-06T01:42:21.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:42:21.442 DEBUG:teuthology.orchestra.run.smithi027:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-06T01:42:21.463 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T01:42:21.622 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.621+0000 7fa5aecd7700 1 Processor -- start 2023-10-06T01:42:21.622 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.621+0000 7fa5aecd7700 1 -- start start 2023-10-06T01:42:21.625 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.622+0000 7fa5aecd7700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa5a8157270 0x7fa5a80ca1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:21.625 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.622+0000 7fa5aecd7700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fa5a80cb920 0x7fa5a8068640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:21.625 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.622+0000 7fa5aecd7700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5a80cb490 con 0x7fa5a8157640 2023-10-06T01:42:21.625 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.622+0000 7fa5aecd7700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5a80ca880 con 0x7fa5a80cb920 2023-10-06T01:42:21.625 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.622+0000 7fa5aecd7700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a80cab60 con 0x7fa5a8157270 2023-10-06T01:42:21.626 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.622+0000 7fa5aca73700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa5a8157270 0x7fa5a80ca1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:21.626 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.622+0000 7fa5ad274700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fa5a80cb920 0x7fa5a8068640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:21.626 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.622+0000 7fa5aca73700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa5a8157270 0x7fa5a80ca1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.27:33002/0 (socket says 172.21.15.27:33002) 2023-10-06T01:42:21.626 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.622+0000 7fa5a7fff700 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7fa5a8157640 0x7fa5a80ae550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.27:40020/0 (socket says 172.21.15.27:40020) 2023-10-06T01:42:21.626 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.622+0000 7fa5ad274700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fa5a80cb920 0x7fa5a8068640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.27:58790/0 (socket says 172.21.15.27:58790) 2023-10-06T01:42:21.626 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.622+0000 7fa5aca73700 1 -- 172.21.15.27:0/709837471 learned_addr learned my addr 172.21.15.27:0/709837471 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:21.626 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.622+0000 7fa5a77fe700 1 -- 172.21.15.27:0/709837471 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3836067782 0 0) 0x7fa5a80cb490 con 0x7fa5a8157640 2023-10-06T01:42:21.626 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.622+0000 7fa5a77fe700 1 -- 172.21.15.27:0/709837471 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa598003570 con 0x7fa5a8157640 2023-10-06T01:42:21.626 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.623+0000 7fa5ad274700 1 -- 172.21.15.27:0/709837471 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa5a8157270 msgr2=0x7fa5a80ca1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:21.626 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.623+0000 7fa5ad274700 1 --2- 172.21.15.27:0/709837471 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa5a8157270 0x7fa5a80ca1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:21.626 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.623+0000 7fa5ad274700 1 -- 172.21.15.27:0/709837471 >> v1:172.21.15.124:6789/0 conn(0x7fa5a8157640 legacy=0x7fa5a80ae550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:21.627 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.623+0000 7fa5ad274700 1 -- 172.21.15.27:0/709837471 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5a80cae40 con 0x7fa5a80cb920 2023-10-06T01:42:21.627 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.623+0000 7fa5aca73700 1 --2- 172.21.15.27:0/709837471 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa5a8157270 0x7fa5a80ca1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:42:21.627 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.623+0000 7fa5ad274700 1 --2- 172.21.15.27:0/709837471 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa5a80cb920 0x7fa5a8068640 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0009e60 tx=0x7fa5a0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a77e365e00539e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:21.627 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.623+0000 7fa5a77fe700 1 -- 172.21.15.27:0/709837471 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a000e070 con 0x7fa5a80cb920 2023-10-06T01:42:21.627 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.623+0000 7fa5a77fe700 1 -- 172.21.15.27:0/709837471 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a000bab0 con 0x7fa5a80cb920 2023-10-06T01:42:21.627 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.623+0000 7fa5a77fe700 1 -- 172.21.15.27:0/709837471 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a000be70 con 0x7fa5a80cb920 2023-10-06T01:42:21.627 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.623+0000 7fa5aecd7700 1 -- 172.21.15.27:0/709837471 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa5a80cb920 msgr2=0x7fa5a8068640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:21.627 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.623+0000 7fa5aecd7700 1 --2- 172.21.15.27:0/709837471 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa5a80cb920 0x7fa5a8068640 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa5a0009e60 tx=0x7fa5a0009780 comp rx=0 tx=0).stop 2023-10-06T01:42:21.627 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.623+0000 7fa5aecd7700 1 -- 172.21.15.27:0/709837471 shutdown_connections 2023-10-06T01:42:21.627 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.623+0000 7fa5aecd7700 1 --2- 172.21.15.27:0/709837471 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa5a80cb920 0x7fa5a8068640 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:21.627 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.623+0000 7fa5aecd7700 1 --2- 172.21.15.27:0/709837471 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa5a8157270 0x7fa5a80ca1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:21.628 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.623+0000 7fa5aecd7700 1 -- 172.21.15.27:0/709837471 >> 172.21.15.27:0/709837471 conn(0x7fa5a80ccd30 msgr2=0x7fa5a80694b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:21.628 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.623+0000 7fa5aecd7700 1 -- 172.21.15.27:0/709837471 shutdown_connections 2023-10-06T01:42:21.628 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.624+0000 7fa5aecd7700 1 -- 172.21.15.27:0/709837471 wait complete. 2023-10-06T01:42:21.630 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.625+0000 7fa5aecd7700 1 Processor -- start 2023-10-06T01:42:21.632 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.625+0000 7fa5aecd7700 1 -- start start 2023-10-06T01:42:21.632 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.625+0000 7fa5aecd7700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa5a80cb920 0x7fa5a81470e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:21.632 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.625+0000 7fa5aecd7700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5a8157270 0x7fa5a816ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:21.632 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.625+0000 7fa5aecd7700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa5a8157640 0x7fa5a806e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:21.632 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.625+0000 7fa5aecd7700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a81493a0 con 0x7fa5a8157270 2023-10-06T01:42:21.632 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.625+0000 7fa5aecd7700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a8148d90 con 0x7fa5a8157640 2023-10-06T01:42:21.632 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.625+0000 7fa5aecd7700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a816b680 con 0x7fa5a80cb920 2023-10-06T01:42:21.632 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.626+0000 7fa5a7fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5a8157270 0x7fa5a816ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:21.632 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.626+0000 7fa5ad274700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa5a8157640 0x7fa5a806e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:21.633 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.626+0000 7fa5ad274700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa5a8157640 0x7fa5a806e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.27:58794/0 (socket says 172.21.15.27:58794) 2023-10-06T01:42:21.633 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.626+0000 7fa5aca73700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa5a80cb920 0x7fa5a81470e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:21.633 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.626+0000 7fa5ad274700 1 -- 172.21.15.27:0/3524891789 learned_addr learned my addr 172.21.15.27:0/3524891789 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:21.633 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.626+0000 7fa5ad274700 1 -- 172.21.15.27:0/3524891789 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa5a80cb920 msgr2=0x7fa5a81470e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:21.633 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.626+0000 7fa5ad274700 1 --2- 172.21.15.27:0/3524891789 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa5a80cb920 0x7fa5a81470e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:21.633 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.626+0000 7fa5ad274700 1 -- 172.21.15.27:0/3524891789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5a8157270 msgr2=0x7fa5a816ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:21.633 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.626+0000 7fa5ad274700 1 --2- 172.21.15.27:0/3524891789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5a8157270 0x7fa5a816ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:21.633 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.626+0000 7fa5ad274700 1 -- 172.21.15.27:0/3524891789 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa59c0094f0 con 0x7fa5a8157640 2023-10-06T01:42:21.633 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.626+0000 7fa5ad274700 1 --2- 172.21.15.27:0/3524891789 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa5a8157640 0x7fa5a806e420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa5a80cd270 tx=0x7fa5a0019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:21.633 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.626+0000 7fa5a4ff9700 1 -- 172.21.15.27:0/3524891789 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0012e10 con 0x7fa5a8157640 2023-10-06T01:42:21.633 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.626+0000 7fa5a4ff9700 1 -- 172.21.15.27:0/3524891789 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a0027470 con 0x7fa5a8157640 2023-10-06T01:42:21.633 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.626+0000 7fa5aecd7700 1 -- 172.21.15.27:0/3524891789 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5a0009a20 con 0x7fa5a8157640 2023-10-06T01:42:21.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.626+0000 7fa5a4ff9700 1 -- 172.21.15.27:0/3524891789 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0027830 con 0x7fa5a8157640 2023-10-06T01:42:21.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.626+0000 7fa5aecd7700 1 -- 172.21.15.27:0/3524891789 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5a8149030 con 0x7fa5a8157640 2023-10-06T01:42:21.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.627+0000 7fa5aecd7700 1 -- 172.21.15.27:0/3524891789 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5a8149510 con 0x7fa5a8157640 2023-10-06T01:42:21.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.628+0000 7fa5a4ff9700 1 -- 172.21.15.27:0/3524891789 <== mon.1 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa5a0024070 con 0x7fa5a8157640 2023-10-06T01:42:21.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.628+0000 7fa5a4ff9700 1 --2- 172.21.15.27:0/3524891789 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa58c066f30 0x7fa58c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:21.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.628+0000 7fa5a4ff9700 1 -- 172.21.15.27:0/3524891789 <== mon.1 v2:172.21.15.124:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6453+0+0 (secure 0 0 0) 0x7fa5a0087110 con 0x7fa5a8157640 2023-10-06T01:42:21.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.628+0000 7fa5aca73700 1 --2- 172.21.15.27:0/3524891789 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa58c066f30 0x7fa58c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:21.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.629+0000 7fa5aca73700 1 --2- 172.21.15.27:0/3524891789 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa58c066f30 0x7fa58c0693e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa59c006d20 tx=0x7fa59c009250 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:21.635 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.633+0000 7fa5a4ff9700 1 -- 172.21.15.27:0/3524891789 <== mon.1 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa5a0054090 con 0x7fa5a8157640 2023-10-06T01:42:21.870 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.869+0000 7fa5aecd7700 1 -- 172.21.15.27:0/3524891789 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5a8064050 con 0x7fa5a8157640 2023-10-06T01:42:21.871 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.870+0000 7fa5a4ff9700 1 -- 172.21.15.27:0/3524891789 <== mon.1 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4599 (secure 0 0 0) 0x7fa5a00576b0 con 0x7fa5a8157640 2023-10-06T01:42:21.871 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:21.871 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4870,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6836/1043044736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":1043044736},{"type":"v1","addr":"172.21.15.124:6836","nonce":1043044736}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4962,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/1207987117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1207987117},{"type":"v1","addr":"172.21.15.27:6834","nonce":1207987117}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4965,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/943261840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":943261840},{"type":"v1","addr":"172.21.15.27:6835","nonce":943261840}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T01:42:03.036784+0000","modified":"2023-10-06T01:42:17.363571+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4974},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4974":{"gid":4974,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.124:6837/325247112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6835","nonce":325247112},{"type":"v1","addr":"172.21.15.124:6837","nonce":325247112}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6,7],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-10-06T01:42:21.873 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.872+0000 7fa5aecd7700 1 -- 172.21.15.27:0/3524891789 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa58c066f30 msgr2=0x7fa58c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:21.874 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.872+0000 7fa5aecd7700 1 --2- 172.21.15.27:0/3524891789 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa58c066f30 0x7fa58c0693e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa59c006d20 tx=0x7fa59c009250 comp rx=0 tx=0).stop 2023-10-06T01:42:21.874 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.872+0000 7fa5aecd7700 1 -- 172.21.15.27:0/3524891789 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa5a8157640 msgr2=0x7fa5a806e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:21.874 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.872+0000 7fa5aecd7700 1 --2- 172.21.15.27:0/3524891789 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa5a8157640 0x7fa5a806e420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa5a80cd270 tx=0x7fa5a0019000 comp rx=0 tx=0).stop 2023-10-06T01:42:21.874 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.872+0000 7fa5aecd7700 1 -- 172.21.15.27:0/3524891789 shutdown_connections 2023-10-06T01:42:21.874 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.872+0000 7fa5aecd7700 1 --2- 172.21.15.27:0/3524891789 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa58c066f30 0x7fa58c0693e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:21.874 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.872+0000 7fa5aecd7700 1 --2- 172.21.15.27:0/3524891789 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa5a80cb920 0x7fa5a81470e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:21.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.872+0000 7fa5aecd7700 1 --2- 172.21.15.27:0/3524891789 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa5a8157640 0x7fa5a806e420 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:21.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.872+0000 7fa5aecd7700 1 --2- 172.21.15.27:0/3524891789 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa5a8157270 0x7fa5a816ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:21.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.872+0000 7fa5aecd7700 1 -- 172.21.15.27:0/3524891789 >> 172.21.15.27:0/3524891789 conn(0x7fa5a80ccd30 msgr2=0x7fa5a8151170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:21.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.872+0000 7fa5aecd7700 1 -- 172.21.15.27:0/3524891789 shutdown_connections 2023-10-06T01:42:21.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:21.872+0000 7fa5aecd7700 1 -- 172.21.15.27:0/3524891789 wait complete. 2023-10-06T01:42:21.875 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 16 2023-10-06T01:42:21.885 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T01:42:22.004 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.002+0000 7f2d13411700 1 Processor -- start 2023-10-06T01:42:22.004 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.002+0000 7f2d13411700 1 -- start start 2023-10-06T01:42:22.006 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.003+0000 7f2d13411700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2d0c1596c0 0x7f2d0c15d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:22.007 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.003+0000 7f2d13411700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2d0c0b09a0 0x7f2d0c0c8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:22.007 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.003+0000 7f2d13411700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d0c15e3a0 con 0x7f2d0c15d960 2023-10-06T01:42:22.007 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.003+0000 7f2d13411700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d0c0cc470 con 0x7f2d0c0b09a0 2023-10-06T01:42:22.007 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.003+0000 7f2d13411700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d0c15dde0 con 0x7f2d0c1596c0 2023-10-06T01:42:22.007 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.003+0000 7f2d109ac700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2d0c0b09a0 0x7f2d0c0c8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:22.007 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.003+0000 7f2d119ae700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2d0c15d960 0x7f2d0c068fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40424/0 (socket says 172.21.15.27:40424) 2023-10-06T01:42:22.007 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.003+0000 7f2d109ac700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2d0c0b09a0 0x7f2d0c0c8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60044/0 (socket says 172.21.15.27:60044) 2023-10-06T01:42:22.007 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.003+0000 7f2d119ae700 1 -- 172.21.15.27:0/2229002243 learned_addr learned my addr 172.21.15.27:0/2229002243 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:22.007 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.004+0000 7f2d111ad700 1 --2- 172.21.15.27:0/2229002243 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2d0c1596c0 0x7f2d0c15d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:22.007 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.004+0000 7f2d109ac700 1 -- 172.21.15.27:0/2229002243 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2d0c1596c0 msgr2=0x7f2d0c15d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:22.007 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.004+0000 7f2d109ac700 1 --2- 172.21.15.27:0/2229002243 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2d0c1596c0 0x7f2d0c15d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:22.008 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.004+0000 7f2d109ac700 1 -- 172.21.15.27:0/2229002243 >> v1:172.21.15.27:6789/0 conn(0x7f2d0c15d960 legacy=0x7f2d0c068fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:22.008 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.004+0000 7f2d109ac700 1 -- 172.21.15.27:0/2229002243 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d0c15e0c0 con 0x7f2d0c0b09a0 2023-10-06T01:42:22.008 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.004+0000 7f2d0bfff700 1 -- 172.21.15.27:0/2229002243 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 810467175 0 0) 0x7f2d0c15e3a0 con 0x7f2d0c15d960 2023-10-06T01:42:22.008 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.004+0000 7f2d109ac700 1 --2- 172.21.15.27:0/2229002243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d0c0b09a0 0x7f2d0c0c8a40 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f2d000098e0 tx=0x7f2d000091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47d3fd213dd05ed0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:22.008 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.004+0000 7f2d0bfff700 1 -- 172.21.15.27:0/2229002243 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d00009e80 con 0x7f2d0c0b09a0 2023-10-06T01:42:22.008 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.004+0000 7f2d0bfff700 1 -- 172.21.15.27:0/2229002243 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d0000b860 con 0x7f2d0c0b09a0 2023-10-06T01:42:22.008 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.004+0000 7f2d0bfff700 1 -- 172.21.15.27:0/2229002243 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d0000bbe0 con 0x7f2d0c0b09a0 2023-10-06T01:42:22.008 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.005+0000 7f2d13411700 1 -- 172.21.15.27:0/2229002243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d0c0b09a0 msgr2=0x7f2d0c0c8a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:22.008 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.005+0000 7f2d13411700 1 --2- 172.21.15.27:0/2229002243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d0c0b09a0 0x7f2d0c0c8a40 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f2d000098e0 tx=0x7f2d000091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:22.008 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.005+0000 7f2d13411700 1 -- 172.21.15.27:0/2229002243 shutdown_connections 2023-10-06T01:42:22.008 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.005+0000 7f2d13411700 1 --2- 172.21.15.27:0/2229002243 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2d0c1596c0 0x7f2d0c15d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:22.008 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.005+0000 7f2d13411700 1 --2- 172.21.15.27:0/2229002243 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d0c0b09a0 0x7f2d0c0c8a40 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:22.009 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.005+0000 7f2d13411700 1 -- 172.21.15.27:0/2229002243 >> 172.21.15.27:0/2229002243 conn(0x7f2d0c067630 msgr2=0x7f2d0c067a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:22.009 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.005+0000 7f2d13411700 1 -- 172.21.15.27:0/2229002243 shutdown_connections 2023-10-06T01:42:22.009 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.005+0000 7f2d13411700 1 -- 172.21.15.27:0/2229002243 wait complete. 2023-10-06T01:42:22.011 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.006+0000 7f2d13411700 1 Processor -- start 2023-10-06T01:42:22.011 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.006+0000 7f2d13411700 1 -- start start 2023-10-06T01:42:22.011 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.007+0000 7f2d13411700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d0c1596c0 0x7f2d0c15afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:22.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.007+0000 7f2d13411700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2d0c06c110 0x7f2d0c16cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:22.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.007+0000 7f2d13411700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2d0c06c580 0x7f2d0c16d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:22.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.007+0000 7f2d13411700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d0c0c7c30 con 0x7f2d0c1596c0 2023-10-06T01:42:22.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.007+0000 7f2d13411700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d0c16deb0 con 0x7f2d0c06c580 2023-10-06T01:42:22.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.007+0000 7f2d13411700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d0c0c8490 con 0x7f2d0c06c110 2023-10-06T01:42:22.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.007+0000 7f2d111ad700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d0c1596c0 0x7f2d0c15afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:22.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.007+0000 7f2d111ad700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d0c1596c0 0x7f2d0c15afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60048/0 (socket says 172.21.15.27:60048) 2023-10-06T01:42:22.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.007+0000 7f2d111ad700 1 -- 172.21.15.27:0/4277863204 learned_addr learned my addr 172.21.15.27:0/4277863204 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:22.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.007+0000 7f2d109ac700 1 --2- 172.21.15.27:0/4277863204 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2d0c06c110 0x7f2d0c16cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:22.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.007+0000 7f2d119ae700 1 --2- 172.21.15.27:0/4277863204 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2d0c06c580 0x7f2d0c16d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:22.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.007+0000 7f2d111ad700 1 -- 172.21.15.27:0/4277863204 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2d0c06c110 msgr2=0x7f2d0c16cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:22.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.007+0000 7f2d111ad700 1 --2- 172.21.15.27:0/4277863204 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2d0c06c110 0x7f2d0c16cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:22.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.007+0000 7f2d111ad700 1 -- 172.21.15.27:0/4277863204 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2d0c06c580 msgr2=0x7f2d0c16d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:22.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.007+0000 7f2d111ad700 1 --2- 172.21.15.27:0/4277863204 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2d0c06c580 0x7f2d0c16d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:22.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.007+0000 7f2d111ad700 1 -- 172.21.15.27:0/4277863204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d00009420 con 0x7f2d0c1596c0 2023-10-06T01:42:22.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.007+0000 7f2d111ad700 1 --2- 172.21.15.27:0/4277863204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d0c1596c0 0x7f2d0c15afb0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f2d0400e1c0 tx=0x7f2d04009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:22.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.007+0000 7f2d097fa700 1 -- 172.21.15.27:0/4277863204 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d0400ed00 con 0x7f2d0c1596c0 2023-10-06T01:42:22.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.008+0000 7f2d097fa700 1 -- 172.21.15.27:0/4277863204 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d0400ee60 con 0x7f2d0c1596c0 2023-10-06T01:42:22.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.008+0000 7f2d13411700 1 -- 172.21.15.27:0/4277863204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d0c16dbd0 con 0x7f2d0c1596c0 2023-10-06T01:42:22.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.008+0000 7f2d097fa700 1 -- 172.21.15.27:0/4277863204 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d04014670 con 0x7f2d0c1596c0 2023-10-06T01:42:22.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.008+0000 7f2d13411700 1 -- 172.21.15.27:0/4277863204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d0c0c8120 con 0x7f2d0c1596c0 2023-10-06T01:42:22.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.009+0000 7f2d13411700 1 -- 172.21.15.27:0/4277863204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d0c06c950 con 0x7f2d0c1596c0 2023-10-06T01:42:22.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.009+0000 7f2d097fa700 1 -- 172.21.15.27:0/4277863204 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2d04014a60 con 0x7f2d0c1596c0 2023-10-06T01:42:22.014 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.009+0000 7f2d097fa700 1 --2- 172.21.15.27:0/4277863204 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2cf8066f60 0x7f2cf8069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:22.014 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.010+0000 7f2d097fa700 1 -- 172.21.15.27:0/4277863204 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f2d04084e90 con 0x7f2d0c1596c0 2023-10-06T01:42:22.014 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.010+0000 7f2d109ac700 1 --2- 172.21.15.27:0/4277863204 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2cf8066f60 0x7f2cf8069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:22.014 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.010+0000 7f2d109ac700 1 --2- 172.21.15.27:0/4277863204 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2cf8066f60 0x7f2cf8069410 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2d000091e0 tx=0x7f2d00017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:22.015 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.014+0000 7f2d097fa700 1 -- 172.21.15.27:0/4277863204 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2d0c06c950 con 0x7f2d0c1596c0 2023-10-06T01:42:22.208 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.207+0000 7f2d13411700 1 -- 172.21.15.27:0/4277863204 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d0c06c950 con 0x7f2d0c1596c0 2023-10-06T01:42:22.209 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.208+0000 7f2d097fa700 1 -- 172.21.15.27:0/4277863204 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+18425 (secure 0 0 0) 0x7f2d0c06c950 con 0x7f2d0c1596c0 2023-10-06T01:42:22.210 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:22.210 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":34,"fsid":"20a7bff0-74f8-4f2f-b6c6-899e804bc965","created":"2023-10-06T01:40:54.474215+0000","modified":"2023-10-06T01:42:18.365024+0000","last_up_change":"2023-10-06T01:41:05.242847+0000","last_in_change":"2023-10-06T01:40:56.890960+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-10-06T01:41:05.685386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-10-06T01:41:06.092995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-10-06T01:42:01.295053+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-10-06T01:42:02.005152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":7,"pool_name":"data_new","create_time":"2023-10-06T01:42:16.105156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b902c86d-800c-40c5-b152-b545e50d02c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":90644},{"type":"v1","addr":"172.21.15.27:6801","nonce":90644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":90644},{"type":"v1","addr":"172.21.15.27:6803","nonce":90644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":90644},{"type":"v1","addr":"172.21.15.27:6807","nonce":90644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":90644},{"type":"v1","addr":"172.21.15.27:6805","nonce":90644}]},"public_addr":"172.21.15.27:6801/90644","cluster_addr":"172.21.15.27:6803/90644","heartbeat_back_addr":"172.21.15.27:6807/90644","heartbeat_front_addr":"172.21.15.27:6805/90644","state":["exists","up"]},{"osd":1,"uuid":"5b4a70e5-b06e-48cd-bfc4-7962c5d05386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":90643},{"type":"v1","addr":"172.21.15.27:6809","nonce":90643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":90643},{"type":"v1","addr":"172.21.15.27:6811","nonce":90643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":90643},{"type":"v1","addr":"172.21.15.27:6815","nonce":90643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":90643},{"type":"v1","addr":"172.21.15.27:6813","nonce":90643}]},"public_addr":"172.21.15.27:6809/90643","cluster_addr":"172.21.15.27:6811/90643","heartbeat_back_addr":"172.21.15.27:6815/90643","heartbeat_front_addr":"172.21.15.27:6813/90643","state":["exists","up"]},{"osd":2,"uuid":"22d9de46-4073-4336-9122-5227e601eca0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":90645},{"type":"v1","addr":"172.21.15.27:6817","nonce":90645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":90645},{"type":"v1","addr":"172.21.15.27:6819","nonce":90645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":90645},{"type":"v1","addr":"172.21.15.27:6823","nonce":90645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":90645},{"type":"v1","addr":"172.21.15.27:6821","nonce":90645}]},"public_addr":"172.21.15.27:6817/90645","cluster_addr":"172.21.15.27:6819/90645","heartbeat_back_addr":"172.21.15.27:6823/90645","heartbeat_front_addr":"172.21.15.27:6821/90645","state":["exists","up"]},{"osd":3,"uuid":"438bb2d3-34e2-4b7c-9734-8360c1b67650","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":90668},{"type":"v1","addr":"172.21.15.27:6825","nonce":90668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":90668},{"type":"v1","addr":"172.21.15.27:6827","nonce":90668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":90668},{"type":"v1","addr":"172.21.15.27:6831","nonce":90668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":90668},{"type":"v1","addr":"172.21.15.27:6829","nonce":90668}]},"public_addr":"172.21.15.27:6825/90668","cluster_addr":"172.21.15.27:6827/90668","heartbeat_back_addr":"172.21.15.27:6831/90668","heartbeat_front_addr":"172.21.15.27:6829/90668","state":["exists","up"]},{"osd":4,"uuid":"e1956a8e-e0f8-4e7c-acbb-40037415a354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":90863},{"type":"v1","addr":"172.21.15.124:6801","nonce":90863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":90863},{"type":"v1","addr":"172.21.15.124:6803","nonce":90863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":90863},{"type":"v1","addr":"172.21.15.124:6807","nonce":90863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":90863},{"type":"v1","addr":"172.21.15.124:6805","nonce":90863}]},"public_addr":"172.21.15.124:6801/90863","cluster_addr":"172.21.15.124:6803/90863","heartbeat_back_addr":"172.21.15.124:6807/90863","heartbeat_front_addr":"172.21.15.124:6805/90863","state":["exists","up"]},{"osd":5,"uuid":"306c59d4-846d-4b73-b1c9-051bec9e9368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":90885},{"type":"v1","addr":"172.21.15.124:6817","nonce":90885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":90885},{"type":"v1","addr":"172.21.15.124:6819","nonce":90885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":90885},{"type":"v1","addr":"172.21.15.124:6823","nonce":90885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":90885},{"type":"v1","addr":"172.21.15.124:6821","nonce":90885}]},"public_addr":"172.21.15.124:6817/90885","cluster_addr":"172.21.15.124:6819/90885","heartbeat_back_addr":"172.21.15.124:6823/90885","heartbeat_front_addr":"172.21.15.124:6821/90885","state":["exists","up"]},{"osd":6,"uuid":"15463e53-ac8d-47d1-b4f6-c0d31f29e4a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":90864},{"type":"v1","addr":"172.21.15.124:6809","nonce":90864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":90864},{"type":"v1","addr":"172.21.15.124:6811","nonce":90864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":90864},{"type":"v1","addr":"172.21.15.124:6815","nonce":90864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":90864},{"type":"v1","addr":"172.21.15.124:6813","nonce":90864}]},"public_addr":"172.21.15.124:6809/90864","cluster_addr":"172.21.15.124:6811/90864","heartbeat_back_addr":"172.21.15.124:6815/90864","heartbeat_front_addr":"172.21.15.124:6813/90864","state":["exists","up"]},{"osd":7,"uuid":"ae463a90-a9d3-498f-86fa-47a35b3fe889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":90894},{"type":"v1","addr":"172.21.15.124:6825","nonce":90894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":90894},{"type":"v1","addr":"172.21.15.124:6827","nonce":90894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":90894},{"type":"v1","addr":"172.21.15.124:6831","nonce":90894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":90894},{"type":"v1","addr":"172.21.15.124:6829","nonce":90894}]},"public_addr":"172.21.15.124:6825/90894","cluster_addr":"172.21.15.124:6827/90894","heartbeat_back_addr":"172.21.15.124:6831/90894","heartbeat_front_addr":"172.21.15.124:6829/90894","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.827704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.839951+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.032230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.050605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.987908+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.031290+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.080707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.053692+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T01:42:22.212 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.210+0000 7f2d13411700 1 -- 172.21.15.27:0/4277863204 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2cf8066f60 msgr2=0x7f2cf8069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:22.212 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.210+0000 7f2d13411700 1 --2- 172.21.15.27:0/4277863204 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2cf8066f60 0x7f2cf8069410 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2d000091e0 tx=0x7f2d00017000 comp rx=0 tx=0).stop 2023-10-06T01:42:22.212 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.210+0000 7f2d13411700 1 -- 172.21.15.27:0/4277863204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d0c1596c0 msgr2=0x7f2d0c15afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:22.212 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.210+0000 7f2d13411700 1 --2- 172.21.15.27:0/4277863204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d0c1596c0 0x7f2d0c15afb0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f2d0400e1c0 tx=0x7f2d04009920 comp rx=0 tx=0).stop 2023-10-06T01:42:22.212 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.210+0000 7f2d13411700 1 -- 172.21.15.27:0/4277863204 shutdown_connections 2023-10-06T01:42:22.212 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.210+0000 7f2d13411700 1 --2- 172.21.15.27:0/4277863204 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2cf8066f60 0x7f2cf8069410 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:22.212 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.210+0000 7f2d13411700 1 --2- 172.21.15.27:0/4277863204 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2d0c06c580 0x7f2d0c16d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:22.213 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.210+0000 7f2d13411700 1 --2- 172.21.15.27:0/4277863204 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d0c1596c0 0x7f2d0c15afb0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:22.213 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.210+0000 7f2d13411700 1 --2- 172.21.15.27:0/4277863204 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2d0c06c110 0x7f2d0c16cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:22.213 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.210+0000 7f2d13411700 1 -- 172.21.15.27:0/4277863204 >> 172.21.15.27:0/4277863204 conn(0x7f2d0c067630 msgr2=0x7f2d0c158910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:22.213 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.211+0000 7f2d13411700 1 -- 172.21.15.27:0/4277863204 shutdown_connections 2023-10-06T01:42:22.213 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.211+0000 7f2d13411700 1 -- 172.21.15.27:0/4277863204 wait complete. 2023-10-06T01:42:22.224 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-10-06T01:41:05.685386+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-10-06T01:41:06.092995+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-06T01:42:01.295053+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '30', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2023-10-06T01:42:02.005152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '31', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}, {'pool': 7, 'pool_name': 'data_new', 'create_time': '2023-10-06T01:42:16.105156+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '34', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-10-06T01:42:22.224 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-10-06T01:42:22.345 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.344+0000 7fc66c899700 1 Processor -- start 2023-10-06T01:42:22.346 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.344+0000 7fc66c899700 1 -- start start 2023-10-06T01:42:22.348 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.345+0000 7fc66c899700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc6680ca400 0x7fc6680cc230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:22.348 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.345+0000 7fc66c899700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fc6680cc770 0x7fc6680c9350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:22.348 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.345+0000 7fc66c899700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc6680ca7d0 0x7fc6680c9890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:22.348 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.345+0000 7fc66c899700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6680c3ce0 con 0x7fc6680ca400 2023-10-06T01:42:22.349 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.345+0000 7fc66c899700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc668069360 con 0x7fc6680cc770 2023-10-06T01:42:22.349 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.345+0000 7fc66c899700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6680c30d0 con 0x7fc6680ca7d0 2023-10-06T01:42:22.349 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.345+0000 7fc6677fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc6680ca400 0x7fc6680cc230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:22.349 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.345+0000 7fc6677fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc6680ca400 0x7fc6680cc230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60062/0 (socket says 172.21.15.27:60062) 2023-10-06T01:42:22.349 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.345+0000 7fc6677fe700 1 -- 172.21.15.27:0/1456601037 learned_addr learned my addr 172.21.15.27:0/1456601037 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:22.349 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.345+0000 7fc667fff700 1 --2- 172.21.15.27:0/1456601037 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc6680ca7d0 0x7fc6680c9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:22.349 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.345+0000 7fc666ffd700 1 --2- 172.21.15.27:0/1456601037 >> v2:172.21.15.124:3300/0 conn(0x7fc6680cc770 0x7fc6680c9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:22.349 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.345+0000 7fc6677fe700 1 -- 172.21.15.27:0/1456601037 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc6680ca7d0 msgr2=0x7fc6680c9890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:22.350 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.345+0000 7fc6677fe700 1 --2- 172.21.15.27:0/1456601037 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc6680ca7d0 0x7fc6680c9890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:22.350 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.345+0000 7fc6677fe700 1 -- 172.21.15.27:0/1456601037 >> v2:172.21.15.124:3300/0 conn(0x7fc6680cc770 msgr2=0x7fc6680c9350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:22.350 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.345+0000 7fc6677fe700 1 --2- 172.21.15.27:0/1456601037 >> v2:172.21.15.124:3300/0 conn(0x7fc6680cc770 0x7fc6680c9350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:22.350 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.345+0000 7fc6677fe700 1 -- 172.21.15.27:0/1456601037 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6680c33b0 con 0x7fc6680ca400 2023-10-06T01:42:22.350 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.346+0000 7fc6677fe700 1 --2- 172.21.15.27:0/1456601037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6680ca400 0x7fc6680cc230 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fc668069dd0 tx=0x7fc658009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b445b0d1c5a358cf server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:22.350 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.346+0000 7fc6667fc700 1 -- 172.21.15.27:0/1456601037 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65800b640 con 0x7fc6680ca400 2023-10-06T01:42:22.350 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.346+0000 7fc6667fc700 1 -- 172.21.15.27:0/1456601037 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc65800b7a0 con 0x7fc6680ca400 2023-10-06T01:42:22.350 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.346+0000 7fc6667fc700 1 -- 172.21.15.27:0/1456601037 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65800bb60 con 0x7fc6680ca400 2023-10-06T01:42:22.351 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.346+0000 7fc66c899700 1 -- 172.21.15.27:0/1456601037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6680ca400 msgr2=0x7fc6680cc230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:22.351 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.346+0000 7fc66c899700 1 --2- 172.21.15.27:0/1456601037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6680ca400 0x7fc6680cc230 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fc668069dd0 tx=0x7fc658009250 comp rx=0 tx=0).stop 2023-10-06T01:42:22.351 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.346+0000 7fc66c899700 1 -- 172.21.15.27:0/1456601037 shutdown_connections 2023-10-06T01:42:22.351 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.346+0000 7fc66c899700 1 --2- 172.21.15.27:0/1456601037 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6680ca400 0x7fc6680cc230 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:22.351 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.346+0000 7fc66c899700 1 --2- 172.21.15.27:0/1456601037 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc6680ca7d0 0x7fc6680c9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:22.351 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.346+0000 7fc66c899700 1 --2- 172.21.15.27:0/1456601037 >> v2:172.21.15.124:3300/0 conn(0x7fc6680cc770 0x7fc6680c9350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:22.351 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.346+0000 7fc66c899700 1 -- 172.21.15.27:0/1456601037 >> 172.21.15.27:0/1456601037 conn(0x7fc6680ce330 msgr2=0x7fc6680ce730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:22.351 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.346+0000 7fc66c899700 1 -- 172.21.15.27:0/1456601037 shutdown_connections 2023-10-06T01:42:22.352 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.346+0000 7fc66c899700 1 -- 172.21.15.27:0/1456601037 wait complete. 2023-10-06T01:42:22.354 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.348+0000 7fc66c899700 1 Processor -- start 2023-10-06T01:42:22.354 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.348+0000 7fc66c899700 1 -- start start 2023-10-06T01:42:22.355 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.348+0000 7fc66c899700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6680ca400 0x7fc66815ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:22.355 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.348+0000 7fc66c899700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc6680ca7d0 0x7fc668140bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:22.355 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.348+0000 7fc66c899700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc6680cc770 0x7fc668153240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:22.355 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.348+0000 7fc66c899700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc668142010 con 0x7fc6680ca400 2023-10-06T01:42:22.355 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.348+0000 7fc66c899700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc66815f230 con 0x7fc6680cc770 2023-10-06T01:42:22.355 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.348+0000 7fc66c899700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc668141400 con 0x7fc6680ca7d0 2023-10-06T01:42:22.355 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.348+0000 7fc6677fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6680ca400 0x7fc66815ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:22.355 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.349+0000 7fc6677fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6680ca400 0x7fc66815ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60070/0 (socket says 172.21.15.27:60070) 2023-10-06T01:42:22.356 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.349+0000 7fc6677fe700 1 -- 172.21.15.27:0/630350890 learned_addr learned my addr 172.21.15.27:0/630350890 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:22.356 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.349+0000 7fc666ffd700 1 --2- 172.21.15.27:0/630350890 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc6680ca7d0 0x7fc668140bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:22.356 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.349+0000 7fc667fff700 1 --2- 172.21.15.27:0/630350890 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc6680cc770 0x7fc668153240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:22.356 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.349+0000 7fc6677fe700 1 -- 172.21.15.27:0/630350890 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc6680ca7d0 msgr2=0x7fc668140bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:22.356 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.349+0000 7fc6677fe700 1 --2- 172.21.15.27:0/630350890 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc6680ca7d0 0x7fc668140bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:22.356 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.349+0000 7fc6677fe700 1 -- 172.21.15.27:0/630350890 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc6680cc770 msgr2=0x7fc668153240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:22.356 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.349+0000 7fc6677fe700 1 --2- 172.21.15.27:0/630350890 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc6680cc770 0x7fc668153240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:22.356 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.349+0000 7fc6677fe700 1 -- 172.21.15.27:0/630350890 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6580094f0 con 0x7fc6680ca400 2023-10-06T01:42:22.356 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.349+0000 7fc6677fe700 1 --2- 172.21.15.27:0/630350890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6680ca400 0x7fc66815ecf0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fc668071360 tx=0x7fc658017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:22.357 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.349+0000 7fc64f7fe700 1 -- 172.21.15.27:0/630350890 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc65800b910 con 0x7fc6680ca400 2023-10-06T01:42:22.357 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.349+0000 7fc64f7fe700 1 -- 172.21.15.27:0/630350890 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc65800ba70 con 0x7fc6680ca400 2023-10-06T01:42:22.357 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.349+0000 7fc66c899700 1 -- 172.21.15.27:0/630350890 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6681416e0 con 0x7fc6680ca400 2023-10-06T01:42:22.357 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.349+0000 7fc64f7fe700 1 -- 172.21.15.27:0/630350890 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc658011630 con 0x7fc6680ca400 2023-10-06T01:42:22.357 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.349+0000 7fc66c899700 1 -- 172.21.15.27:0/630350890 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc668141ca0 con 0x7fc6680ca400 2023-10-06T01:42:22.357 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.350+0000 7fc66c899700 1 -- 172.21.15.27:0/630350890 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc66804c3c0 con 0x7fc6680ca400 2023-10-06T01:42:22.357 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.351+0000 7fc64f7fe700 1 -- 172.21.15.27:0/630350890 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc6580119a0 con 0x7fc6680ca400 2023-10-06T01:42:22.357 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.351+0000 7fc64f7fe700 1 --2- 172.21.15.27:0/630350890 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc654066f30 0x7fc6540693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:22.357 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.351+0000 7fc64f7fe700 1 -- 172.21.15.27:0/630350890 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6453+0+0 (secure 0 0 0) 0x7fc658085f60 con 0x7fc6680ca400 2023-10-06T01:42:22.357 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.351+0000 7fc666ffd700 1 --2- 172.21.15.27:0/630350890 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc654066f30 0x7fc6540693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:22.358 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.352+0000 7fc666ffd700 1 --2- 172.21.15.27:0/630350890 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc654066f30 0x7fc6540693e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc6680c68a0 tx=0x7fc65c0091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:22.358 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.355+0000 7fc64f7fe700 1 -- 172.21.15.27:0/630350890 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc658052ee0 con 0x7fc6680ca400 2023-10-06T01:42:22.549 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.547+0000 7fc66c899700 1 -- 172.21.15.27:0/630350890 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fc6680cf150 con 0x7fc6680ca400 2023-10-06T01:42:22.549 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.548+0000 7fc64f7fe700 1 -- 172.21.15.27:0/630350890 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v34) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc658056500 con 0x7fc6680ca400 2023-10-06T01:42:22.549 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 8 2023-10-06T01:42:22.551 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.550+0000 7fc66c899700 1 -- 172.21.15.27:0/630350890 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc654066f30 msgr2=0x7fc6540693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:22.551 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.550+0000 7fc66c899700 1 --2- 172.21.15.27:0/630350890 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc654066f30 0x7fc6540693e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc6680c68a0 tx=0x7fc65c0091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:22.552 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.550+0000 7fc66c899700 1 -- 172.21.15.27:0/630350890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6680ca400 msgr2=0x7fc66815ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:22.552 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.550+0000 7fc66c899700 1 --2- 172.21.15.27:0/630350890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6680ca400 0x7fc66815ecf0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fc668071360 tx=0x7fc658017000 comp rx=0 tx=0).stop 2023-10-06T01:42:22.552 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.550+0000 7fc66c899700 1 -- 172.21.15.27:0/630350890 shutdown_connections 2023-10-06T01:42:22.552 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.550+0000 7fc66c899700 1 --2- 172.21.15.27:0/630350890 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc654066f30 0x7fc6540693e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:22.552 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.550+0000 7fc66c899700 1 --2- 172.21.15.27:0/630350890 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc6680cc770 0x7fc668153240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:22.552 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.550+0000 7fc66c899700 1 --2- 172.21.15.27:0/630350890 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc6680ca400 0x7fc66815ecf0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:22.552 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.550+0000 7fc66c899700 1 --2- 172.21.15.27:0/630350890 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc6680ca7d0 0x7fc668140bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:22.552 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.550+0000 7fc66c899700 1 -- 172.21.15.27:0/630350890 >> 172.21.15.27:0/630350890 conn(0x7fc6680ce330 msgr2=0x7fc66806b1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:22.552 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.550+0000 7fc66c899700 1 -- 172.21.15.27:0/630350890 shutdown_connections 2023-10-06T01:42:22.552 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.550+0000 7fc66c899700 1 -- 172.21.15.27:0/630350890 wait complete. 2023-10-06T01:42:22.563 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-06T01:42:22.685 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.683+0000 7f97bcbb5700 1 Processor -- start 2023-10-06T01:42:22.685 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.684+0000 7f97bcbb5700 1 -- start start 2023-10-06T01:42:22.687 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.684+0000 7f97bcbb5700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f97b8159500 0x7f97b81598d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:22.687 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.685+0000 7f97bcbb5700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f97b815dab0 0x7f97b80cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:22.688 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.685+0000 7f97bcbb5700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97b80b1110 con 0x7f97b80b3fd0 2023-10-06T01:42:22.688 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.685+0000 7f97bcbb5700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97b80b13b0 con 0x7f97b815dab0 2023-10-06T01:42:22.688 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.685+0000 7f97bcbb5700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97b80b1690 con 0x7f97b8159500 2023-10-06T01:42:22.688 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.685+0000 7f97b6d9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f97b815dab0 0x7f97b80cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:22.688 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.685+0000 7f97b6d9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f97b815dab0 0x7f97b80cf6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60076/0 (socket says 172.21.15.27:60076) 2023-10-06T01:42:22.688 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.685+0000 7f97b6d9d700 1 -- 172.21.15.27:0/3572120987 learned_addr learned my addr 172.21.15.27:0/3572120987 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:22.688 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.685+0000 7f97b659c700 1 --2- 172.21.15.27:0/3572120987 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f97b8159500 0x7f97b81598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:22.688 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.685+0000 7f97b6d9d700 1 -- 172.21.15.27:0/3572120987 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f97b8159500 msgr2=0x7f97b81598d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:22.688 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.685+0000 7f97b6d9d700 1 --2- 172.21.15.27:0/3572120987 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f97b8159500 0x7f97b81598d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:22.689 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.685+0000 7f97b6d9d700 1 -- 172.21.15.27:0/3572120987 >> v1:172.21.15.124:6789/0 conn(0x7f97b80b3fd0 legacy=0x7f97b815d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:22.689 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.685+0000 7f97b6d9d700 1 -- 172.21.15.27:0/3572120987 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b80b1970 con 0x7f97b815dab0 2023-10-06T01:42:22.689 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.685+0000 7f97b6d9d700 1 --2- 172.21.15.27:0/3572120987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97b815dab0 0x7f97b80cf6c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f97a8009f20 tx=0x7f97a8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36683bf910d27c15 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:22.689 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.685+0000 7f97b559a700 1 -- 172.21.15.27:0/3572120987 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a800e070 con 0x7f97b815dab0 2023-10-06T01:42:22.689 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.685+0000 7f97b559a700 1 -- 172.21.15.27:0/3572120987 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97a800b9d0 con 0x7f97b815dab0 2023-10-06T01:42:22.689 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.685+0000 7f97b559a700 1 -- 172.21.15.27:0/3572120987 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a80092e0 con 0x7f97b815dab0 2023-10-06T01:42:22.689 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.686+0000 7f97bcbb5700 1 -- 172.21.15.27:0/3572120987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97b815dab0 msgr2=0x7f97b80cf6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:22.689 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.686+0000 7f97bcbb5700 1 --2- 172.21.15.27:0/3572120987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97b815dab0 0x7f97b80cf6c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f97a8009f20 tx=0x7f97a8009840 comp rx=0 tx=0).stop 2023-10-06T01:42:22.689 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.686+0000 7f97bcbb5700 1 -- 172.21.15.27:0/3572120987 shutdown_connections 2023-10-06T01:42:22.689 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.686+0000 7f97bcbb5700 1 --2- 172.21.15.27:0/3572120987 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97b815dab0 0x7f97b80cf6c0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:22.689 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.686+0000 7f97bcbb5700 1 --2- 172.21.15.27:0/3572120987 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f97b8159500 0x7f97b81598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:22.690 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.686+0000 7f97bcbb5700 1 -- 172.21.15.27:0/3572120987 >> 172.21.15.27:0/3572120987 conn(0x7f97b8067630 msgr2=0x7f97b8069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:22.690 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.686+0000 7f97bcbb5700 1 -- 172.21.15.27:0/3572120987 shutdown_connections 2023-10-06T01:42:22.690 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.686+0000 7f97bcbb5700 1 -- 172.21.15.27:0/3572120987 wait complete. 2023-10-06T01:42:22.693 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.687+0000 7f97bcbb5700 1 Processor -- start 2023-10-06T01:42:22.693 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.687+0000 7f97bcbb5700 1 -- start start 2023-10-06T01:42:22.693 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.688+0000 7f97bcbb5700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f97b80b3fd0 0x7f97b816a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:22.693 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.688+0000 7f97bcbb5700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f97b8159500 0x7f97b8143d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:22.693 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.688+0000 7f97bcbb5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97b815dab0 0x7f97b8068b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:22.693 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.688+0000 7f97bcbb5700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b81459e0 con 0x7f97b815dab0 2023-10-06T01:42:22.693 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.688+0000 7f97bcbb5700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b816aab0 con 0x7f97b8159500 2023-10-06T01:42:22.693 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.688+0000 7f97bcbb5700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97b8144af0 con 0x7f97b80b3fd0 2023-10-06T01:42:22.694 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.688+0000 7f97b6d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97b815dab0 0x7f97b8068b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:22.694 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.688+0000 7f97b6d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97b815dab0 0x7f97b8068b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60088/0 (socket says 172.21.15.27:60088) 2023-10-06T01:42:22.694 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.688+0000 7f97b6d9d700 1 -- 172.21.15.27:0/2174408473 learned_addr learned my addr 172.21.15.27:0/2174408473 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:22.694 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.688+0000 7f97b659c700 1 --2- 172.21.15.27:0/2174408473 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f97b80b3fd0 0x7f97b816a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:22.694 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.688+0000 7f97b5d9b700 1 --2- 172.21.15.27:0/2174408473 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f97b8159500 0x7f97b8143d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:22.694 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.688+0000 7f97b6d9d700 1 -- 172.21.15.27:0/2174408473 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f97b80b3fd0 msgr2=0x7f97b816a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:22.694 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.688+0000 7f97b6d9d700 1 --2- 172.21.15.27:0/2174408473 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f97b80b3fd0 0x7f97b816a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:22.694 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.688+0000 7f97b6d9d700 1 -- 172.21.15.27:0/2174408473 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f97b8159500 msgr2=0x7f97b8143d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:22.694 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.688+0000 7f97b6d9d700 1 --2- 172.21.15.27:0/2174408473 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f97b8159500 0x7f97b8143d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:22.694 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.688+0000 7f97b6d9d700 1 -- 172.21.15.27:0/2174408473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a8009ae0 con 0x7f97b815dab0 2023-10-06T01:42:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.688+0000 7f97b5d9b700 1 --2- 172.21.15.27:0/2174408473 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f97b8159500 0x7f97b8143d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:42:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.688+0000 7f97b6d9d700 1 --2- 172.21.15.27:0/2174408473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97b815dab0 0x7f97b8068b80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f97a8012000 tx=0x7f97a8018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.689+0000 7f97ae7fc700 1 -- 172.21.15.27:0/2174408473 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a8008bc0 con 0x7f97b815dab0 2023-10-06T01:42:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.689+0000 7f97ae7fc700 1 -- 172.21.15.27:0/2174408473 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97a8008d60 con 0x7f97b815dab0 2023-10-06T01:42:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.689+0000 7f97bcbb5700 1 -- 172.21.15.27:0/2174408473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97b8144dd0 con 0x7f97b815dab0 2023-10-06T01:42:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.689+0000 7f97ae7fc700 1 -- 172.21.15.27:0/2174408473 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a801c660 con 0x7f97b815dab0 2023-10-06T01:42:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.689+0000 7f97bcbb5700 1 -- 172.21.15.27:0/2174408473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97b8145390 con 0x7f97b815dab0 2023-10-06T01:42:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.690+0000 7f97bcbb5700 1 -- 172.21.15.27:0/2174408473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97b8064050 con 0x7f97b815dab0 2023-10-06T01:42:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.690+0000 7f97ae7fc700 1 -- 172.21.15.27:0/2174408473 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f97a8003040 con 0x7f97b815dab0 2023-10-06T01:42:22.695 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.690+0000 7f97ae7fc700 1 --2- 172.21.15.27:0/2174408473 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9798066f30 0x7f97980693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:22.696 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.691+0000 7f97ae7fc700 1 -- 172.21.15.27:0/2174408473 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f97a8086090 con 0x7f97b815dab0 2023-10-06T01:42:22.696 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.691+0000 7f97b659c700 1 --2- 172.21.15.27:0/2174408473 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9798066f30 0x7f97980693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:22.696 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.691+0000 7f97b659c700 1 --2- 172.21.15.27:0/2174408473 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9798066f30 0x7f97980693e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f97b0009db0 tx=0x7f97b00091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:22.696 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.695+0000 7f97ae7fc700 1 -- 172.21.15.27:0/2174408473 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f97a8053010 con 0x7f97b815dab0 2023-10-06T01:42:22.887 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.885+0000 7f97bcbb5700 1 -- 172.21.15.27:0/2174408473 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f97b80d0420 con 0x7f97b815dab0 2023-10-06T01:42:22.887 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.885+0000 7f97ae7fc700 1 -- 172.21.15.27:0/2174408473 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v34) v1 ==== 93+0+10 (secure 0 0 0) 0x7f97a8056630 con 0x7f97b815dab0 2023-10-06T01:42:22.887 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 1 2023-10-06T01:42:22.889 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.887+0000 7f97bcbb5700 1 -- 172.21.15.27:0/2174408473 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9798066f30 msgr2=0x7f97980693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:22.889 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.887+0000 7f97bcbb5700 1 --2- 172.21.15.27:0/2174408473 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9798066f30 0x7f97980693e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f97b0009db0 tx=0x7f97b00091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:22.889 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.888+0000 7f97bcbb5700 1 -- 172.21.15.27:0/2174408473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97b815dab0 msgr2=0x7f97b8068b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:22.890 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.888+0000 7f97bcbb5700 1 --2- 172.21.15.27:0/2174408473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97b815dab0 0x7f97b8068b80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f97a8012000 tx=0x7f97a8018000 comp rx=0 tx=0).stop 2023-10-06T01:42:22.890 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.888+0000 7f97bcbb5700 1 -- 172.21.15.27:0/2174408473 shutdown_connections 2023-10-06T01:42:22.890 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.888+0000 7f97bcbb5700 1 --2- 172.21.15.27:0/2174408473 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9798066f30 0x7f97980693e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:22.890 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.888+0000 7f97bcbb5700 1 --2- 172.21.15.27:0/2174408473 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f97b80b3fd0 0x7f97b816a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:22.890 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.888+0000 7f97bcbb5700 1 --2- 172.21.15.27:0/2174408473 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f97b815dab0 0x7f97b8068b80 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:22.890 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.888+0000 7f97bcbb5700 1 --2- 172.21.15.27:0/2174408473 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f97b8159500 0x7f97b8143d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:22.890 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.888+0000 7f97bcbb5700 1 -- 172.21.15.27:0/2174408473 >> 172.21.15.27:0/2174408473 conn(0x7f97b8067630 msgr2=0x7f97b80b4440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:22.890 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.888+0000 7f97bcbb5700 1 -- 172.21.15.27:0/2174408473 shutdown_connections 2023-10-06T01:42:22.890 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:22.888+0000 7f97bcbb5700 1 -- 172.21.15.27:0/2174408473 wait complete. 2023-10-06T01:42:22.901 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-06T01:42:23.022 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.021+0000 7f999eb41700 1 Processor -- start 2023-10-06T01:42:23.023 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.021+0000 7f999eb41700 1 -- start start 2023-10-06T01:42:23.025 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.022+0000 7f999eb41700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f9998159500 0x7f99981598d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:23.025 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.022+0000 7f999eb41700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f999815dab0 0x7f99980cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:23.025 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.022+0000 7f999eb41700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99980b1110 con 0x7f99980b3fd0 2023-10-06T01:42:23.025 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.022+0000 7f999eb41700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99980b13b0 con 0x7f999815dab0 2023-10-06T01:42:23.025 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.022+0000 7f999eb41700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99980b1690 con 0x7f9998159500 2023-10-06T01:42:23.025 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.022+0000 7f999d0de700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f999815dab0 0x7f99980cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:23.025 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.022+0000 7f999d0de700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f999815dab0 0x7f99980cf6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60104/0 (socket says 172.21.15.27:60104) 2023-10-06T01:42:23.026 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.022+0000 7f999d0de700 1 -- 172.21.15.27:0/290122525 learned_addr learned my addr 172.21.15.27:0/290122525 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:23.026 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.022+0000 7f9997fff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f99980b3fd0 0x7f999815d3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40434/0 (socket says 172.21.15.27:40434) 2023-10-06T01:42:23.026 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.022+0000 7f999c8dd700 1 --2- 172.21.15.27:0/290122525 >> v2:172.21.15.124:3300/0 conn(0x7f9998159500 0x7f99981598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:23.026 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.022+0000 7f999d0de700 1 -- 172.21.15.27:0/290122525 >> v2:172.21.15.124:3300/0 conn(0x7f9998159500 msgr2=0x7f99981598d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:23.026 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.022+0000 7f999d0de700 1 --2- 172.21.15.27:0/290122525 >> v2:172.21.15.124:3300/0 conn(0x7f9998159500 0x7f99981598d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:23.026 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.022+0000 7f999d0de700 1 -- 172.21.15.27:0/290122525 >> v1:172.21.15.27:6789/0 conn(0x7f99980b3fd0 legacy=0x7f999815d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:23.026 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.022+0000 7f999d0de700 1 -- 172.21.15.27:0/290122525 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99980b1970 con 0x7f999815dab0 2023-10-06T01:42:23.026 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.022+0000 7f99977fe700 1 -- 172.21.15.27:0/290122525 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2637704470 0 0) 0x7f99980b1110 con 0x7f99980b3fd0 2023-10-06T01:42:23.027 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.023+0000 7f999d0de700 1 --2- 172.21.15.27:0/290122525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f999815dab0 0x7f99980cf6c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f9998063e00 tx=0x7f9990009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d624bd76b39e4a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:23.027 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.023+0000 7f99977fe700 1 -- 172.21.15.27:0/290122525 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999000e070 con 0x7f999815dab0 2023-10-06T01:42:23.027 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.023+0000 7f99977fe700 1 -- 172.21.15.27:0/290122525 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f999000ba50 con 0x7f999815dab0 2023-10-06T01:42:23.027 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.023+0000 7f99977fe700 1 -- 172.21.15.27:0/290122525 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99900092e0 con 0x7f999815dab0 2023-10-06T01:42:23.027 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.023+0000 7f999eb41700 1 -- 172.21.15.27:0/290122525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f999815dab0 msgr2=0x7f99980cf6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:23.027 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.023+0000 7f999eb41700 1 --2- 172.21.15.27:0/290122525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f999815dab0 0x7f99980cf6c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f9998063e00 tx=0x7f9990009840 comp rx=0 tx=0).stop 2023-10-06T01:42:23.027 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.023+0000 7f999eb41700 1 -- 172.21.15.27:0/290122525 shutdown_connections 2023-10-06T01:42:23.027 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.023+0000 7f999eb41700 1 --2- 172.21.15.27:0/290122525 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f999815dab0 0x7f99980cf6c0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:23.027 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.023+0000 7f999eb41700 1 --2- 172.21.15.27:0/290122525 >> v2:172.21.15.124:3300/0 conn(0x7f9998159500 0x7f99981598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:23.027 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.023+0000 7f999eb41700 1 -- 172.21.15.27:0/290122525 >> 172.21.15.27:0/290122525 conn(0x7f9998067630 msgr2=0x7f9998069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:23.028 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.023+0000 7f999eb41700 1 -- 172.21.15.27:0/290122525 shutdown_connections 2023-10-06T01:42:23.028 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.023+0000 7f999eb41700 1 -- 172.21.15.27:0/290122525 wait complete. 2023-10-06T01:42:23.030 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.024+0000 7f999eb41700 1 Processor -- start 2023-10-06T01:42:23.030 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.025+0000 7f999eb41700 1 -- start start 2023-10-06T01:42:23.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.025+0000 7f999eb41700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f99980b3fd0 0x7f9998144080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:23.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.025+0000 7f999eb41700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9998159500 0x7f9998144c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:23.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.025+0000 7f999eb41700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f999815dab0 0x7f9998145160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:23.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.025+0000 7f999eb41700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99981550d0 con 0x7f999815dab0 2023-10-06T01:42:23.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.025+0000 7f999eb41700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9998147cc0 con 0x7f99980b3fd0 2023-10-06T01:42:23.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.025+0000 7f999eb41700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9998148090 con 0x7f9998159500 2023-10-06T01:42:23.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.025+0000 7f999d0de700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f999815dab0 0x7f9998145160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:23.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.025+0000 7f999d0de700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f999815dab0 0x7f9998145160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60116/0 (socket says 172.21.15.27:60116) 2023-10-06T01:42:23.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.025+0000 7f999d0de700 1 -- 172.21.15.27:0/3151326904 learned_addr learned my addr 172.21.15.27:0/3151326904 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:23.032 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.026+0000 7f999c8dd700 1 --2- 172.21.15.27:0/3151326904 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f99980b3fd0 0x7f9998144080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:23.032 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.026+0000 7f9997fff700 1 --2- 172.21.15.27:0/3151326904 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9998159500 0x7f9998144c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:23.032 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.026+0000 7f999d0de700 1 -- 172.21.15.27:0/3151326904 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9998159500 msgr2=0x7f9998144c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:23.032 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.026+0000 7f999d0de700 1 --2- 172.21.15.27:0/3151326904 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9998159500 0x7f9998144c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:23.032 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.026+0000 7f999d0de700 1 -- 172.21.15.27:0/3151326904 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f99980b3fd0 msgr2=0x7f9998144080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:23.032 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.026+0000 7f999d0de700 1 --2- 172.21.15.27:0/3151326904 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f99980b3fd0 0x7f9998144080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:23.032 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.026+0000 7f999d0de700 1 -- 172.21.15.27:0/3151326904 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9990009ae0 con 0x7f999815dab0 2023-10-06T01:42:23.032 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.026+0000 7f999d0de700 1 --2- 172.21.15.27:0/3151326904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f999815dab0 0x7f9998145160 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f9998088d00 tx=0x7f9990018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:23.032 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.026+0000 7f9994ff9700 1 -- 172.21.15.27:0/3151326904 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9990008ce0 con 0x7f999815dab0 2023-10-06T01:42:23.032 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.026+0000 7f9994ff9700 1 -- 172.21.15.27:0/3151326904 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9990008e80 con 0x7f999815dab0 2023-10-06T01:42:23.032 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.026+0000 7f999eb41700 1 -- 172.21.15.27:0/3151326904 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9998154b10 con 0x7f999815dab0 2023-10-06T01:42:23.033 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.026+0000 7f9994ff9700 1 -- 172.21.15.27:0/3151326904 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999001c660 con 0x7f999815dab0 2023-10-06T01:42:23.033 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.026+0000 7f999eb41700 1 -- 172.21.15.27:0/3151326904 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99981445c0 con 0x7f999815dab0 2023-10-06T01:42:23.033 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.027+0000 7f999eb41700 1 -- 172.21.15.27:0/3151326904 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9998147e60 con 0x7f999815dab0 2023-10-06T01:42:23.033 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.028+0000 7f9994ff9700 1 -- 172.21.15.27:0/3151326904 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9990003040 con 0x7f999815dab0 2023-10-06T01:42:23.033 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.028+0000 7f9994ff9700 1 --2- 172.21.15.27:0/3151326904 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9984066f30 0x7f99840693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:23.033 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.028+0000 7f9994ff9700 1 -- 172.21.15.27:0/3151326904 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f9990086080 con 0x7f999815dab0 2023-10-06T01:42:23.033 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.028+0000 7f999c8dd700 1 --2- 172.21.15.27:0/3151326904 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9984066f30 0x7f99840693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:23.033 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.029+0000 7f999c8dd700 1 --2- 172.21.15.27:0/3151326904 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9984066f30 0x7f99840693e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9988006fd0 tx=0x7f9988008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:23.034 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.032+0000 7f9994ff9700 1 -- 172.21.15.27:0/3151326904 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9998147e60 con 0x7f999815dab0 2023-10-06T01:42:23.223 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.221+0000 7f999eb41700 1 -- 172.21.15.27:0/3151326904 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9998147e60 con 0x7f999815dab0 2023-10-06T01:42:23.223 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.222+0000 7f9994ff9700 1 -- 172.21.15.27:0/3151326904 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v34) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9998147e60 con 0x7f999815dab0 2023-10-06T01:42:23.224 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2023-10-06T01:42:23.226 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.224+0000 7f999eb41700 1 -- 172.21.15.27:0/3151326904 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9984066f30 msgr2=0x7f99840693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:23.226 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.224+0000 7f999eb41700 1 --2- 172.21.15.27:0/3151326904 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9984066f30 0x7f99840693e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9988006fd0 tx=0x7f9988008040 comp rx=0 tx=0).stop 2023-10-06T01:42:23.226 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.224+0000 7f999eb41700 1 -- 172.21.15.27:0/3151326904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f999815dab0 msgr2=0x7f9998145160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:23.226 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.224+0000 7f999eb41700 1 --2- 172.21.15.27:0/3151326904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f999815dab0 0x7f9998145160 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f9998088d00 tx=0x7f9990018000 comp rx=0 tx=0).stop 2023-10-06T01:42:23.226 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.224+0000 7f999eb41700 1 -- 172.21.15.27:0/3151326904 shutdown_connections 2023-10-06T01:42:23.226 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.224+0000 7f999eb41700 1 --2- 172.21.15.27:0/3151326904 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9984066f30 0x7f99840693e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:23.226 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.224+0000 7f999eb41700 1 --2- 172.21.15.27:0/3151326904 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f999815dab0 0x7f9998145160 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:23.226 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.224+0000 7f999eb41700 1 --2- 172.21.15.27:0/3151326904 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f99980b3fd0 0x7f9998144080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:23.226 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.224+0000 7f999eb41700 1 --2- 172.21.15.27:0/3151326904 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9998159500 0x7f9998144c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:23.227 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.224+0000 7f999eb41700 1 -- 172.21.15.27:0/3151326904 >> 172.21.15.27:0/3151326904 conn(0x7f9998067630 msgr2=0x7f99980cfbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:23.227 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.224+0000 7f999eb41700 1 -- 172.21.15.27:0/3151326904 shutdown_connections 2023-10-06T01:42:23.227 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.224+0000 7f999eb41700 1 -- 172.21.15.27:0/3151326904 wait complete. 2023-10-06T01:42:23.237 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-06T01:42:23.358 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.357+0000 7fd32c854700 1 Processor -- start 2023-10-06T01:42:23.359 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.357+0000 7fd32c854700 1 -- start start 2023-10-06T01:42:23.361 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.358+0000 7fd32c854700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd32815dab0 0x7fd3280cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:23.361 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.358+0000 7fd32c854700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3280b1110 con 0x7fd328159500 2023-10-06T01:42:23.361 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.358+0000 7fd32c854700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3280b13b0 con 0x7fd3281598d0 2023-10-06T01:42:23.361 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.358+0000 7fd32c854700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3280b1690 con 0x7fd32815dab0 2023-10-06T01:42:23.361 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.358+0000 7fd32659c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd328159500 0x7fd3280b3fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40444/0 (socket says 172.21.15.27:40444) 2023-10-06T01:42:23.362 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.358+0000 7fd32659c700 1 -- 172.21.15.27:0/2290870735 learned_addr learned my addr 172.21.15.27:0/2290870735 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:23.362 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.358+0000 7fd326d9d700 1 --2- 172.21.15.27:0/2290870735 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd32815dab0 0x7fd3280cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:23.362 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.358+0000 7fd32559a700 1 -- 172.21.15.27:0/2290870735 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2973120955 0 0) 0x7fd3280b1110 con 0x7fd328159500 2023-10-06T01:42:23.362 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.358+0000 7fd32559a700 1 -- 172.21.15.27:0/2290870735 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd314003570 con 0x7fd328159500 2023-10-06T01:42:23.362 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.359+0000 7fd32559a700 1 -- 172.21.15.27:0/2290870735 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1356761051 0 0) 0x7fd3280b13b0 con 0x7fd3281598d0 2023-10-06T01:42:23.362 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.359+0000 7fd32559a700 1 -- 172.21.15.27:0/2290870735 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3280b1110 con 0x7fd3281598d0 2023-10-06T01:42:23.362 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.359+0000 7fd326d9d700 1 -- 172.21.15.27:0/2290870735 >> v1:172.21.15.124:6789/0 conn(0x7fd3281598d0 legacy=0x7fd32815d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:23.362 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.359+0000 7fd326d9d700 1 -- 172.21.15.27:0/2290870735 >> v1:172.21.15.27:6789/0 conn(0x7fd328159500 legacy=0x7fd3280b3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:23.362 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.359+0000 7fd326d9d700 1 -- 172.21.15.27:0/2290870735 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3280b1970 con 0x7fd32815dab0 2023-10-06T01:42:23.362 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.359+0000 7fd326d9d700 1 --2- 172.21.15.27:0/2290870735 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd32815dab0 0x7fd3280cf6c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd318009910 tx=0x7fd3180091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d703fcb5317e2c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:23.362 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.359+0000 7fd32559a700 1 -- 172.21.15.27:0/2290870735 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd318011630 con 0x7fd32815dab0 2023-10-06T01:42:23.363 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.359+0000 7fd32559a700 1 -- 172.21.15.27:0/2290870735 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd318011790 con 0x7fd32815dab0 2023-10-06T01:42:23.363 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.359+0000 7fd32559a700 1 -- 172.21.15.27:0/2290870735 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd318011b30 con 0x7fd32815dab0 2023-10-06T01:42:23.363 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.359+0000 7fd32c854700 1 -- 172.21.15.27:0/2290870735 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd32815dab0 msgr2=0x7fd3280cf6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:23.363 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.360+0000 7fd32c854700 1 --2- 172.21.15.27:0/2290870735 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd32815dab0 0x7fd3280cf6c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd318009910 tx=0x7fd3180091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:23.363 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.360+0000 7fd32c854700 1 -- 172.21.15.27:0/2290870735 shutdown_connections 2023-10-06T01:42:23.363 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.360+0000 7fd32c854700 1 --2- 172.21.15.27:0/2290870735 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd32815dab0 0x7fd3280cf6c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:23.363 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.360+0000 7fd32c854700 1 -- 172.21.15.27:0/2290870735 >> 172.21.15.27:0/2290870735 conn(0x7fd328067630 msgr2=0x7fd328069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:23.363 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.360+0000 7fd32c854700 1 -- 172.21.15.27:0/2290870735 shutdown_connections 2023-10-06T01:42:23.363 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.360+0000 7fd32c854700 1 -- 172.21.15.27:0/2290870735 wait complete. 2023-10-06T01:42:23.366 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.361+0000 7fd32c854700 1 Processor -- start 2023-10-06T01:42:23.366 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.361+0000 7fd32c854700 1 -- start start 2023-10-06T01:42:23.366 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.361+0000 7fd32c854700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd328159500 0x7fd328152830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:23.366 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.361+0000 7fd32c854700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd3281598d0 0x7fd328150310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:23.366 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.361+0000 7fd32c854700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd32815dab0 0x7fd32815c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:23.366 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.361+0000 7fd32c854700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd328152e90 con 0x7fd328159500 2023-10-06T01:42:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.361+0000 7fd32c854700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd328153170 con 0x7fd3281598d0 2023-10-06T01:42:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.361+0000 7fd32c854700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd32815a660 con 0x7fd32815dab0 2023-10-06T01:42:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.362+0000 7fd32659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd328159500 0x7fd328152830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.362+0000 7fd32659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd328159500 0x7fd328152830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60120/0 (socket says 172.21.15.27:60120) 2023-10-06T01:42:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.362+0000 7fd32659c700 1 -- 172.21.15.27:0/3205694347 learned_addr learned my addr 172.21.15.27:0/3205694347 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.362+0000 7fd325d9b700 1 --2- 172.21.15.27:0/3205694347 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd3281598d0 0x7fd328150310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.362+0000 7fd326d9d700 1 --2- 172.21.15.27:0/3205694347 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd32815dab0 0x7fd32815c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.362+0000 7fd32659c700 1 -- 172.21.15.27:0/3205694347 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd32815dab0 msgr2=0x7fd32815c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.362+0000 7fd32659c700 1 --2- 172.21.15.27:0/3205694347 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd32815dab0 0x7fd32815c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.362+0000 7fd32659c700 1 -- 172.21.15.27:0/3205694347 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd3281598d0 msgr2=0x7fd328150310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.362+0000 7fd32659c700 1 --2- 172.21.15.27:0/3205694347 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd3281598d0 0x7fd328150310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:23.367 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.362+0000 7fd32659c700 1 -- 172.21.15.27:0/3205694347 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd318009420 con 0x7fd328159500 2023-10-06T01:42:23.368 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.362+0000 7fd32659c700 1 --2- 172.21.15.27:0/3205694347 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd328159500 0x7fd328152830 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fd32000cd80 tx=0x7fd320001b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:23.368 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.362+0000 7fd31e7fc700 1 -- 172.21.15.27:0/3205694347 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32000aa80 con 0x7fd328159500 2023-10-06T01:42:23.368 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.362+0000 7fd31e7fc700 1 -- 172.21.15.27:0/3205694347 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd32000ac00 con 0x7fd328159500 2023-10-06T01:42:23.368 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.362+0000 7fd32c854700 1 -- 172.21.15.27:0/3205694347 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd32815a940 con 0x7fd328159500 2023-10-06T01:42:23.368 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.362+0000 7fd31e7fc700 1 -- 172.21.15.27:0/3205694347 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3200070f0 con 0x7fd328159500 2023-10-06T01:42:23.368 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.362+0000 7fd32c854700 1 -- 172.21.15.27:0/3205694347 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd32815ce90 con 0x7fd328159500 2023-10-06T01:42:23.368 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.363+0000 7fd32c854700 1 -- 172.21.15.27:0/3205694347 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd328153390 con 0x7fd328159500 2023-10-06T01:42:23.368 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.364+0000 7fd31e7fc700 1 -- 172.21.15.27:0/3205694347 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd3200074e0 con 0x7fd328159500 2023-10-06T01:42:23.368 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.364+0000 7fd31e7fc700 1 --2- 172.21.15.27:0/3205694347 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd308066f30 0x7fd3080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:23.368 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.364+0000 7fd31e7fc700 1 -- 172.21.15.27:0/3205694347 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6453+0+0 (secure 0 0 0) 0x7fd320084bb0 con 0x7fd328159500 2023-10-06T01:42:23.368 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.364+0000 7fd325d9b700 1 --2- 172.21.15.27:0/3205694347 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd308066f30 0x7fd3080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:23.368 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.365+0000 7fd325d9b700 1 --2- 172.21.15.27:0/3205694347 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd308066f30 0x7fd3080693e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd32814bcc0 tx=0x7fd310007d50 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:23.369 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.368+0000 7fd31e7fc700 1 -- 172.21.15.27:0/3205694347 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd320051bb0 con 0x7fd328159500 2023-10-06T01:42:23.562 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.561+0000 7fd32c854700 1 -- 172.21.15.27:0/3205694347 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd328064050 con 0x7fd328159500 2023-10-06T01:42:23.563 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.561+0000 7fd31e7fc700 1 -- 172.21.15.27:0/3205694347 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v34) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd3200551d0 con 0x7fd328159500 2023-10-06T01:42:23.563 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 64 2023-10-06T01:42:23.565 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.563+0000 7fd32c854700 1 -- 172.21.15.27:0/3205694347 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd308066f30 msgr2=0x7fd3080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:23.565 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.563+0000 7fd32c854700 1 --2- 172.21.15.27:0/3205694347 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd308066f30 0x7fd3080693e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fd32814bcc0 tx=0x7fd310007d50 comp rx=0 tx=0).stop 2023-10-06T01:42:23.565 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.563+0000 7fd32c854700 1 -- 172.21.15.27:0/3205694347 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd328159500 msgr2=0x7fd328152830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:23.565 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.563+0000 7fd32c854700 1 --2- 172.21.15.27:0/3205694347 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd328159500 0x7fd328152830 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fd32000cd80 tx=0x7fd320001b90 comp rx=0 tx=0).stop 2023-10-06T01:42:23.565 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.563+0000 7fd32c854700 1 -- 172.21.15.27:0/3205694347 shutdown_connections 2023-10-06T01:42:23.565 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.563+0000 7fd32c854700 1 --2- 172.21.15.27:0/3205694347 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd308066f30 0x7fd3080693e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:23.566 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.563+0000 7fd32c854700 1 --2- 172.21.15.27:0/3205694347 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd32815dab0 0x7fd32815c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:23.566 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.563+0000 7fd32c854700 1 --2- 172.21.15.27:0/3205694347 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd328159500 0x7fd328152830 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:23.566 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.563+0000 7fd32c854700 1 --2- 172.21.15.27:0/3205694347 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd3281598d0 0x7fd328150310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:23.566 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.563+0000 7fd32c854700 1 -- 172.21.15.27:0/3205694347 >> 172.21.15.27:0/3205694347 conn(0x7fd328067630 msgr2=0x7fd328069a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:23.566 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.563+0000 7fd32c854700 1 -- 172.21.15.27:0/3205694347 shutdown_connections 2023-10-06T01:42:23.566 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.563+0000 7fd32c854700 1 -- 172.21.15.27:0/3205694347 wait complete. 2023-10-06T01:42:23.576 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2023-10-06T01:42:23.697 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.696+0000 7f7b82def700 1 Processor -- start 2023-10-06T01:42:23.698 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.696+0000 7f7b82def700 1 -- start start 2023-10-06T01:42:23.701 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.697+0000 7f7b82def700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7b7c069000 0x7f7b7c159150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:23.701 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.697+0000 7f7b82def700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b7c0694e0 con 0x7f7b7c15d8e0 2023-10-06T01:42:23.701 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.697+0000 7f7b82def700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b7c0697c0 con 0x7f7b7c15d510 2023-10-06T01:42:23.701 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.697+0000 7f7b82def700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7c069aa0 con 0x7f7b7c069000 2023-10-06T01:42:23.701 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.697+0000 7f7b825ee700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7b7c15d8e0 0x7f7b7c0b0c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40452/0 (socket says 172.21.15.27:40452) 2023-10-06T01:42:23.701 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.697+0000 7f7b825ee700 1 -- 172.21.15.27:0/3105792388 learned_addr learned my addr 172.21.15.27:0/3105792388 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:23.701 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.697+0000 7f7b815ec700 1 --2- 172.21.15.27:0/3105792388 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7b7c069000 0x7f7b7c159150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:23.701 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.697+0000 7f7b80deb700 1 -- 172.21.15.27:0/3105792388 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2396598026 0 0) 0x7f7b7c0694e0 con 0x7f7b7c15d8e0 2023-10-06T01:42:23.701 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.697+0000 7f7b80deb700 1 -- 172.21.15.27:0/3105792388 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b64003570 con 0x7f7b7c15d8e0 2023-10-06T01:42:23.702 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.698+0000 7f7b80deb700 1 -- 172.21.15.27:0/3105792388 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2064903970 0 0) 0x7f7b7c0697c0 con 0x7f7b7c15d510 2023-10-06T01:42:23.702 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.698+0000 7f7b80deb700 1 -- 172.21.15.27:0/3105792388 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b7c0694e0 con 0x7f7b7c15d510 2023-10-06T01:42:23.702 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.698+0000 7f7b80deb700 1 -- 172.21.15.27:0/3105792388 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 497174308 0 0) 0x7f7b64003570 con 0x7f7b7c15d8e0 2023-10-06T01:42:23.702 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.698+0000 7f7b80deb700 1 -- 172.21.15.27:0/3105792388 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7b7c069000 msgr2=0x7f7b7c159150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:23.702 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.698+0000 7f7b80deb700 1 --2- 172.21.15.27:0/3105792388 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7b7c069000 0x7f7b7c159150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:23.702 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.698+0000 7f7b80deb700 1 -- 172.21.15.27:0/3105792388 >> v1:172.21.15.124:6789/0 conn(0x7f7b7c15d510 legacy=0x7f7b7c0b3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:23.702 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.698+0000 7f7b80deb700 1 -- 172.21.15.27:0/3105792388 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b7c069d80 con 0x7f7b7c15d8e0 2023-10-06T01:42:23.702 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.698+0000 7f7b80deb700 1 -- 172.21.15.27:0/3105792388 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2619078944 0 0) 0x7f7b6c0017f0 con 0x7f7b7c15d8e0 2023-10-06T01:42:23.702 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.698+0000 7f7b815ec700 1 --2- 172.21.15.27:0/3105792388 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7b7c069000 0x7f7b7c159150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:42:23.702 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.698+0000 7f7b80deb700 1 -- 172.21.15.27:0/3105792388 >> v1:172.21.15.27:6789/0 conn(0x7f7b7c15d8e0 legacy=0x7f7b7c0b0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:23.702 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.698+0000 7f7b80deb700 1 --2- 172.21.15.27:0/3105792388 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7b7c069000 0x7f7b64003c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:23.703 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.698+0000 7f7b80deb700 1 --2- 172.21.15.27:0/3105792388 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7b64004160 0x7f7b64004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:23.703 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.698+0000 7f7b80deb700 1 --2- 172.21.15.27:0/3105792388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b64004d50 0x7f7b640051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:23.703 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.698+0000 7f7b80deb700 1 -- 172.21.15.27:0/3105792388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7c069aa0 con 0x7f7b64004d50 2023-10-06T01:42:23.703 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.698+0000 7f7b80deb700 1 -- 172.21.15.27:0/3105792388 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6400bba0 con 0x7f7b64004160 2023-10-06T01:42:23.703 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.698+0000 7f7b80deb700 1 -- 172.21.15.27:0/3105792388 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b6400bd60 con 0x7f7b7c069000 2023-10-06T01:42:23.703 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.698+0000 7f7b825ee700 1 --2- 172.21.15.27:0/3105792388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b64004d50 0x7f7b640051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:23.703 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.698+0000 7f7b815ec700 1 --2- 172.21.15.27:0/3105792388 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7b64004160 0x7f7b64004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:23.703 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.698+0000 7f7b81ded700 1 --2- 172.21.15.27:0/3105792388 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7b7c069000 0x7f7b64003c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:23.703 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.698+0000 7f7b825ee700 1 -- 172.21.15.27:0/3105792388 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7b7c069000 msgr2=0x7f7b64003c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:23.703 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.698+0000 7f7b825ee700 1 --2- 172.21.15.27:0/3105792388 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7b7c069000 0x7f7b64003c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:23.703 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.698+0000 7f7b825ee700 1 -- 172.21.15.27:0/3105792388 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7b64004160 msgr2=0x7f7b64004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:23.703 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.698+0000 7f7b825ee700 1 --2- 172.21.15.27:0/3105792388 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7b64004160 0x7f7b64004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:23.703 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.698+0000 7f7b825ee700 1 -- 172.21.15.27:0/3105792388 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b70009420 con 0x7f7b64004d50 2023-10-06T01:42:23.704 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.698+0000 7f7b825ee700 1 --2- 172.21.15.27:0/3105792388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b64004d50 0x7f7b640051c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c00a8d0 tx=0x7f7b6c002190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9507d066144254 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:23.704 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.699+0000 7f7b80deb700 1 -- 172.21.15.27:0/3105792388 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c0070d0 con 0x7f7b64004d50 2023-10-06T01:42:23.704 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.699+0000 7f7b80deb700 1 -- 172.21.15.27:0/3105792388 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b6c007230 con 0x7f7b64004d50 2023-10-06T01:42:23.704 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.699+0000 7f7b80deb700 1 -- 172.21.15.27:0/3105792388 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c0075f0 con 0x7f7b64004d50 2023-10-06T01:42:23.704 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.699+0000 7f7b82def700 1 -- 172.21.15.27:0/3105792388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b64004d50 msgr2=0x7f7b640051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:23.704 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.699+0000 7f7b82def700 1 --2- 172.21.15.27:0/3105792388 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b64004d50 0x7f7b640051c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c00a8d0 tx=0x7f7b6c002190 comp rx=0 tx=0).stop 2023-10-06T01:42:23.704 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.699+0000 7f7b825ee700 1 -- 172.21.15.27:0/3105792388 reap_dead start 2023-10-06T01:42:23.704 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.699+0000 7f7b82def700 1 -- 172.21.15.27:0/3105792388 shutdown_connections 2023-10-06T01:42:23.704 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.699+0000 7f7b82def700 1 -- 172.21.15.27:0/3105792388 >> 172.21.15.27:0/3105792388 conn(0x7f7b7c0ccfa0 msgr2=0x7f7b7c0cd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:23.704 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.699+0000 7f7b82def700 1 -- 172.21.15.27:0/3105792388 shutdown_connections 2023-10-06T01:42:23.704 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.699+0000 7f7b82def700 1 -- 172.21.15.27:0/3105792388 wait complete. 2023-10-06T01:42:23.707 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.700+0000 7f7b82def700 1 Processor -- start 2023-10-06T01:42:23.707 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.700+0000 7f7b82def700 1 -- start start 2023-10-06T01:42:23.707 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.701+0000 7f7b82def700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7b7c153020 0x7f7b7c15b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:23.707 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.701+0000 7f7b82def700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b7c15bc90 0x7f7b7c15c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:23.707 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.701+0000 7f7b82def700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7b7c16d910 0x7f7b7c16dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:23.707 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.701+0000 7f7b82def700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7c15c640 con 0x7f7b7c15bc90 2023-10-06T01:42:23.707 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.701+0000 7f7b82def700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7c0c6af0 con 0x7f7b7c153020 2023-10-06T01:42:23.707 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.701+0000 7f7b82def700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7c0c5c00 con 0x7f7b7c16d910 2023-10-06T01:42:23.708 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.701+0000 7f7b815ec700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b7c15bc90 0x7f7b7c15c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:23.708 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.701+0000 7f7b815ec700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b7c15bc90 0x7f7b7c15c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60140/0 (socket says 172.21.15.27:60140) 2023-10-06T01:42:23.708 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.701+0000 7f7b815ec700 1 -- 172.21.15.27:0/3685907419 learned_addr learned my addr 172.21.15.27:0/3685907419 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:23.708 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.701+0000 7f7b81ded700 1 --2- 172.21.15.27:0/3685907419 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7b7c153020 0x7f7b7c15b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:23.708 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.701+0000 7f7b825ee700 1 --2- 172.21.15.27:0/3685907419 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7b7c16d910 0x7f7b7c16dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:23.708 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.701+0000 7f7b815ec700 1 -- 172.21.15.27:0/3685907419 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7b7c16d910 msgr2=0x7f7b7c16dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:23.708 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.701+0000 7f7b815ec700 1 --2- 172.21.15.27:0/3685907419 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7b7c16d910 0x7f7b7c16dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:23.708 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.701+0000 7f7b815ec700 1 -- 172.21.15.27:0/3685907419 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7b7c153020 msgr2=0x7f7b7c15b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:23.708 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.701+0000 7f7b815ec700 1 --2- 172.21.15.27:0/3685907419 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7b7c153020 0x7f7b7c15b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:23.708 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.702+0000 7f7b815ec700 1 -- 172.21.15.27:0/3685907419 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b7c0c5ee0 con 0x7f7b7c15bc90 2023-10-06T01:42:23.708 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.702+0000 7f7b825ee700 1 --2- 172.21.15.27:0/3685907419 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7b7c16d910 0x7f7b7c16dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:42:23.708 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.702+0000 7f7b815ec700 1 --2- 172.21.15.27:0/3685907419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b7c15bc90 0x7f7b7c15c100 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f7b70011d40 tx=0x7f7b700091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:23.709 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.702+0000 7f7b79ffb700 1 -- 172.21.15.27:0/3685907419 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7000fcd0 con 0x7f7b7c15bc90 2023-10-06T01:42:23.709 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.702+0000 7f7b79ffb700 1 -- 172.21.15.27:0/3685907419 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b7000fe30 con 0x7f7b7c15bc90 2023-10-06T01:42:23.709 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.702+0000 7f7b79ffb700 1 -- 172.21.15.27:0/3685907419 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b70014670 con 0x7f7b7c15bc90 2023-10-06T01:42:23.709 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.702+0000 7f7b82def700 1 -- 172.21.15.27:0/3685907419 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b7c0c61c0 con 0x7f7b7c15bc90 2023-10-06T01:42:23.709 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.702+0000 7f7b82def700 1 -- 172.21.15.27:0/3685907419 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b7c0c6780 con 0x7f7b7c15bc90 2023-10-06T01:42:23.709 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.703+0000 7f7b82def700 1 -- 172.21.15.27:0/3685907419 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b7c0c6ca0 con 0x7f7b7c15bc90 2023-10-06T01:42:23.709 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.703+0000 7f7b79ffb700 1 -- 172.21.15.27:0/3685907419 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7b70014a60 con 0x7f7b7c15bc90 2023-10-06T01:42:23.709 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.704+0000 7f7b79ffb700 1 --2- 172.21.15.27:0/3685907419 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7b68066f30 0x7f7b680693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:23.709 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.704+0000 7f7b79ffb700 1 -- 172.21.15.27:0/3685907419 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f7b70018030 con 0x7f7b7c15bc90 2023-10-06T01:42:23.709 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.704+0000 7f7b81ded700 1 --2- 172.21.15.27:0/3685907419 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7b68066f30 0x7f7b680693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:23.709 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.704+0000 7f7b81ded700 1 --2- 172.21.15.27:0/3685907419 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7b68066f30 0x7f7b680693e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7b74007e40 tx=0x7f7b74009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:23.710 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.708+0000 7f7b79ffb700 1 -- 172.21.15.27:0/3685907419 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7b7c0c6ca0 con 0x7f7b7c15bc90 2023-10-06T01:42:23.903 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.902+0000 7f7b82def700 1 -- 172.21.15.27:0/3685907419 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) v1 -- 0x7f7b7c0c6c60 con 0x7f7b7c15bc90 2023-10-06T01:42:23.904 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.902+0000 7f7b79ffb700 1 -- 172.21.15.27:0/3685907419 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v34) v1 ==== 97+0+11 (secure 0 0 0) 0x7f7b70052400 con 0x7f7b7c15bc90 2023-10-06T01:42:23.904 INFO:teuthology.orchestra.run.smithi027.stdout:pg_num: 32 2023-10-06T01:42:23.906 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.904+0000 7f7b82def700 1 -- 172.21.15.27:0/3685907419 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7b68066f30 msgr2=0x7f7b680693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:23.906 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.904+0000 7f7b82def700 1 --2- 172.21.15.27:0/3685907419 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7b68066f30 0x7f7b680693e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7b74007e40 tx=0x7f7b74009040 comp rx=0 tx=0).stop 2023-10-06T01:42:23.906 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.904+0000 7f7b82def700 1 -- 172.21.15.27:0/3685907419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b7c15bc90 msgr2=0x7f7b7c15c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:23.906 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.904+0000 7f7b82def700 1 --2- 172.21.15.27:0/3685907419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b7c15bc90 0x7f7b7c15c100 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f7b70011d40 tx=0x7f7b700091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:23.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.904+0000 7f7b82def700 1 -- 172.21.15.27:0/3685907419 shutdown_connections 2023-10-06T01:42:23.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.904+0000 7f7b82def700 1 --2- 172.21.15.27:0/3685907419 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7b68066f30 0x7f7b680693e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:23.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.904+0000 7f7b82def700 1 --2- 172.21.15.27:0/3685907419 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7b7c16d910 0x7f7b7c16dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:23.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.904+0000 7f7b82def700 1 --2- 172.21.15.27:0/3685907419 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7b7c153020 0x7f7b7c15b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:23.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.904+0000 7f7b82def700 1 --2- 172.21.15.27:0/3685907419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7b7c15bc90 0x7f7b7c15c100 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:23.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.904+0000 7f7b82def700 1 -- 172.21.15.27:0/3685907419 >> 172.21.15.27:0/3685907419 conn(0x7f7b7c0ccfa0 msgr2=0x7f7b7c153820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:23.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.905+0000 7f7b82def700 1 -- 172.21.15.27:0/3685907419 shutdown_connections 2023-10-06T01:42:23.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:23.905+0000 7f7b82def700 1 -- 172.21.15.27:0/3685907419 wait complete. 2023-10-06T01:42:23.918 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T01:42:24.037 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.035+0000 7f70e0028700 1 Processor -- start 2023-10-06T01:42:24.037 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.035+0000 7f70e0028700 1 -- start start 2023-10-06T01:42:24.039 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.036+0000 7f70e0028700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f70d8157570 0x7f70d80b4060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:24.039 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.036+0000 7f70e0028700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70d81580d0 con 0x7f70d815b210 2023-10-06T01:42:24.039 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.036+0000 7f70e0028700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70d815ba40 con 0x7f70d80add70 2023-10-06T01:42:24.040 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.036+0000 7f70e0028700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70d8158c70 con 0x7f70d8157570 2023-10-06T01:42:24.040 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.036+0000 7f70dd5c3700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f70d815b210 0x7f70d8158560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40464/0 (socket says 172.21.15.27:40464) 2023-10-06T01:42:24.041 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.036+0000 7f70dd5c3700 1 -- 172.21.15.27:0/1991198202 learned_addr learned my addr 172.21.15.27:0/1991198202 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:24.041 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.036+0000 7f70dddc4700 1 --2- 172.21.15.27:0/1991198202 >> v2:172.21.15.124:3300/0 conn(0x7f70d8157570 0x7f70d80b4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:24.042 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.037+0000 7f70dcdc2700 1 -- 172.21.15.27:0/1991198202 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1152477011 0 0) 0x7f70d81580d0 con 0x7f70d815b210 2023-10-06T01:42:24.042 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.037+0000 7f70dcdc2700 1 -- 172.21.15.27:0/1991198202 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70c0003570 con 0x7f70d815b210 2023-10-06T01:42:24.042 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.037+0000 7f70dddc4700 1 -- 172.21.15.27:0/1991198202 >> v1:172.21.15.124:6789/0 conn(0x7f70d80add70 legacy=0x7f70d80ae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:24.042 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.037+0000 7f70dcdc2700 1 -- 172.21.15.27:0/1991198202 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1871328770 0 0) 0x7f70d815ba40 con 0x7f70d80add70 2023-10-06T01:42:24.042 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.037+0000 7f70dddc4700 1 -- 172.21.15.27:0/1991198202 >> v1:172.21.15.27:6789/0 conn(0x7f70d815b210 legacy=0x7f70d8158560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:24.042 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.037+0000 7f70dddc4700 1 -- 172.21.15.27:0/1991198202 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70d815bd20 con 0x7f70d8157570 2023-10-06T01:42:24.042 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.037+0000 7f70dddc4700 1 --2- 172.21.15.27:0/1991198202 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f70d8157570 0x7f70d80b4060 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f70d00098e0 tx=0x7f70d00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f4be532423b34fb server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:24.042 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.037+0000 7f70dcdc2700 1 -- 172.21.15.27:0/1991198202 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70cc001470 con 0x7f70d8157570 2023-10-06T01:42:24.042 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.037+0000 7f70dcdc2700 1 -- 172.21.15.27:0/1991198202 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70d0011420 con 0x7f70d8157570 2023-10-06T01:42:24.042 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.037+0000 7f70dcdc2700 1 -- 172.21.15.27:0/1991198202 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d00117c0 con 0x7f70d8157570 2023-10-06T01:42:24.042 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.038+0000 7f70e0028700 1 -- 172.21.15.27:0/1991198202 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f70d8157570 msgr2=0x7f70d80b4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:24.043 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.038+0000 7f70e0028700 1 --2- 172.21.15.27:0/1991198202 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f70d8157570 0x7f70d80b4060 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f70d00098e0 tx=0x7f70d00091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:24.043 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.038+0000 7f70e0028700 1 -- 172.21.15.27:0/1991198202 shutdown_connections 2023-10-06T01:42:24.043 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.038+0000 7f70e0028700 1 --2- 172.21.15.27:0/1991198202 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f70d8157570 0x7f70d80b4060 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:24.043 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.038+0000 7f70e0028700 1 -- 172.21.15.27:0/1991198202 >> 172.21.15.27:0/1991198202 conn(0x7f70d8067630 msgr2=0x7f70d8067a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:24.043 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.038+0000 7f70e0028700 1 -- 172.21.15.27:0/1991198202 shutdown_connections 2023-10-06T01:42:24.043 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.038+0000 7f70e0028700 1 -- 172.21.15.27:0/1991198202 wait complete. 2023-10-06T01:42:24.046 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.039+0000 7f70e0028700 1 Processor -- start 2023-10-06T01:42:24.046 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.039+0000 7f70e0028700 1 -- start start 2023-10-06T01:42:24.046 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.040+0000 7f70e0028700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70d80add70 0x7f70d8152140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:24.046 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.040+0000 7f70e0028700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f70d8157570 0x7f70d80c27e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:24.046 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.040+0000 7f70dddc4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70d80add70 0x7f70d8152140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:24.046 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.040+0000 7f70dddc4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70d80add70 0x7f70d8152140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60144/0 (socket says 172.21.15.27:60144) 2023-10-06T01:42:24.046 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.040+0000 7f70e0028700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f70d815b210 0x7f70d8151530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:24.047 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.040+0000 7f70dddc4700 1 -- 172.21.15.27:0/3098279728 learned_addr learned my addr 172.21.15.27:0/3098279728 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:24.047 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.040+0000 7f70e0028700 1 -- 172.21.15.27:0/3098279728 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d80c2010 con 0x7f70d80add70 2023-10-06T01:42:24.047 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.040+0000 7f70e0028700 1 -- 172.21.15.27:0/3098279728 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70d8151a70 con 0x7f70d8157570 2023-10-06T01:42:24.047 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.040+0000 7f70e0028700 1 -- 172.21.15.27:0/3098279728 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70d81537f0 con 0x7f70d815b210 2023-10-06T01:42:24.047 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.040+0000 7f70dd5c3700 1 --2- 172.21.15.27:0/3098279728 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f70d8157570 0x7f70d80c27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:24.047 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.040+0000 7f70de5c5700 1 --2- 172.21.15.27:0/3098279728 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f70d815b210 0x7f70d8151530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:24.047 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.040+0000 7f70dd5c3700 1 -- 172.21.15.27:0/3098279728 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f70d815b210 msgr2=0x7f70d8151530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:24.047 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.040+0000 7f70dd5c3700 1 --2- 172.21.15.27:0/3098279728 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f70d815b210 0x7f70d8151530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:24.047 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.040+0000 7f70dd5c3700 1 -- 172.21.15.27:0/3098279728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70d80add70 msgr2=0x7f70d8152140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:24.047 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.040+0000 7f70dd5c3700 1 --2- 172.21.15.27:0/3098279728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70d80add70 0x7f70d8152140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:24.047 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.040+0000 7f70dd5c3700 1 -- 172.21.15.27:0/3098279728 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70d0009420 con 0x7f70d8157570 2023-10-06T01:42:24.047 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.040+0000 7f70dddc4700 1 --2- 172.21.15.27:0/3098279728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70d80add70 0x7f70d8152140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:42:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.040+0000 7f70dd5c3700 1 --2- 172.21.15.27:0/3098279728 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f70d8157570 0x7f70d80c27e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f70cc00cb60 tx=0x7f70cc001b00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.040+0000 7f70d5ffb700 1 -- 172.21.15.27:0/3098279728 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70cc007360 con 0x7f70d8157570 2023-10-06T01:42:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.041+0000 7f70d5ffb700 1 -- 172.21.15.27:0/3098279728 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70cc0074e0 con 0x7f70d8157570 2023-10-06T01:42:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.041+0000 7f70e0028700 1 -- 172.21.15.27:0/3098279728 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70d806be80 con 0x7f70d8157570 2023-10-06T01:42:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.041+0000 7f70d5ffb700 1 -- 172.21.15.27:0/3098279728 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70cc004830 con 0x7f70d8157570 2023-10-06T01:42:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.041+0000 7f70e0028700 1 -- 172.21.15.27:0/3098279728 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70d81531a0 con 0x7f70d8157570 2023-10-06T01:42:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.042+0000 7f70e0028700 1 -- 172.21.15.27:0/3098279728 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70d81539a0 con 0x7f70d8157570 2023-10-06T01:42:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.042+0000 7f70d5ffb700 1 -- 172.21.15.27:0/3098279728 <== mon.1 v2:172.21.15.124:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f70cc004c20 con 0x7f70d8157570 2023-10-06T01:42:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.042+0000 7f70d5ffb700 1 --2- 172.21.15.27:0/3098279728 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f70c4066f30 0x7f70c40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.042+0000 7f70d5ffb700 1 -- 172.21.15.27:0/3098279728 <== mon.1 v2:172.21.15.124:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f70cc084080 con 0x7f70d8157570 2023-10-06T01:42:24.048 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.043+0000 7f70dddc4700 1 --2- 172.21.15.27:0/3098279728 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f70c4066f30 0x7f70c40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:24.049 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.043+0000 7f70dddc4700 1 --2- 172.21.15.27:0/3098279728 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f70c4066f30 0x7f70c40693e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f70d00093c0 tx=0x7f70d0017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:24.049 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.047+0000 7f70d5ffb700 1 -- 172.21.15.27:0/3098279728 <== mon.1 v2:172.21.15.124:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f70cc051080 con 0x7f70d8157570 2023-10-06T01:42:24.286 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.285+0000 7f70e0028700 1 -- 172.21.15.27:0/3098279728 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70d8064050 con 0x7f70d8157570 2023-10-06T01:42:24.287 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.287+0000 7f70d5ffb700 1 -- 172.21.15.27:0/3098279728 <== mon.1 v2:172.21.15.124:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4599 (secure 0 0 0) 0x7f70cc0546a0 con 0x7f70d8157570 2023-10-06T01:42:24.287 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:24.287 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4870,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6836/1043044736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":1043044736},{"type":"v1","addr":"172.21.15.124:6836","nonce":1043044736}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4962,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/1207987117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1207987117},{"type":"v1","addr":"172.21.15.27:6834","nonce":1207987117}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4965,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/943261840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":943261840},{"type":"v1","addr":"172.21.15.27:6835","nonce":943261840}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T01:42:03.036784+0000","modified":"2023-10-06T01:42:17.363571+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4974},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4974":{"gid":4974,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.124:6837/325247112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6835","nonce":325247112},{"type":"v1","addr":"172.21.15.124:6837","nonce":325247112}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6,7],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-10-06T01:42:24.290 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.288+0000 7f70e0028700 1 -- 172.21.15.27:0/3098279728 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f70c4066f30 msgr2=0x7f70c40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:24.290 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.288+0000 7f70e0028700 1 --2- 172.21.15.27:0/3098279728 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f70c4066f30 0x7f70c40693e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f70d00093c0 tx=0x7f70d0017000 comp rx=0 tx=0).stop 2023-10-06T01:42:24.290 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.288+0000 7f70e0028700 1 -- 172.21.15.27:0/3098279728 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f70d8157570 msgr2=0x7f70d80c27e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:24.290 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.288+0000 7f70e0028700 1 --2- 172.21.15.27:0/3098279728 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f70d8157570 0x7f70d80c27e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f70cc00cb60 tx=0x7f70cc001b00 comp rx=0 tx=0).stop 2023-10-06T01:42:24.290 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.288+0000 7f70e0028700 1 -- 172.21.15.27:0/3098279728 shutdown_connections 2023-10-06T01:42:24.290 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.288+0000 7f70e0028700 1 --2- 172.21.15.27:0/3098279728 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f70c4066f30 0x7f70c40693e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:24.290 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.288+0000 7f70e0028700 1 --2- 172.21.15.27:0/3098279728 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f70d815b210 0x7f70d8151530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:24.290 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.288+0000 7f70e0028700 1 --2- 172.21.15.27:0/3098279728 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f70d80add70 0x7f70d8152140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:24.290 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.288+0000 7f70e0028700 1 --2- 172.21.15.27:0/3098279728 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f70d8157570 0x7f70d80c27e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:24.290 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.288+0000 7f70e0028700 1 -- 172.21.15.27:0/3098279728 >> 172.21.15.27:0/3098279728 conn(0x7f70d8067630 msgr2=0x7f70d815c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:24.291 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.288+0000 7f70e0028700 1 -- 172.21.15.27:0/3098279728 shutdown_connections 2023-10-06T01:42:24.291 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.288+0000 7f70e0028700 1 -- 172.21.15.27:0/3098279728 wait complete. 2023-10-06T01:42:24.291 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 16 2023-10-06T01:42:24.301 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T01:42:24.422 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.421+0000 7fd3225a4700 1 Processor -- start 2023-10-06T01:42:24.423 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.421+0000 7fd3225a4700 1 -- start start 2023-10-06T01:42:24.425 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.422+0000 7fd3225a4700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd31c157b70 0x7fd31c158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:24.425 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.422+0000 7fd3225a4700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd31c0b2190 0x7fd31c0b25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:24.425 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.422+0000 7fd3225a4700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd31c0cc390 con 0x7fd31c159210 2023-10-06T01:42:24.425 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.422+0000 7fd3225a4700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd31c0b2b10 con 0x7fd31c0b2190 2023-10-06T01:42:24.425 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.422+0000 7fd3225a4700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd31c0cc630 con 0x7fd31c157b70 2023-10-06T01:42:24.425 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.422+0000 7fd321da3700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fd31c0b2190 0x7fd31c0b25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:24.425 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.422+0000 7fd320da1700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fd31c159210 0x7fd31c0cd6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40478/0 (socket says 172.21.15.27:40478) 2023-10-06T01:42:24.426 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.422+0000 7fd320da1700 1 -- 172.21.15.27:0/1897938128 learned_addr learned my addr 172.21.15.27:0/1897938128 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:24.426 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.422+0000 7fd3215a2700 1 --2- 172.21.15.27:0/1897938128 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd31c157b70 0x7fd31c158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:24.426 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.422+0000 7fd321da3700 1 -- 172.21.15.27:0/1897938128 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd31c157b70 msgr2=0x7fd31c158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:24.426 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.422+0000 7fd321da3700 1 --2- 172.21.15.27:0/1897938128 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd31c157b70 0x7fd31c158cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:24.426 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.422+0000 7fd321da3700 1 -- 172.21.15.27:0/1897938128 >> v1:172.21.15.27:6789/0 conn(0x7fd31c159210 legacy=0x7fd31c0cd6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:24.426 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.422+0000 7fd31bfff700 1 -- 172.21.15.27:0/1897938128 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1071560136 0 0) 0x7fd31c0cc390 con 0x7fd31c159210 2023-10-06T01:42:24.426 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.422+0000 7fd321da3700 1 -- 172.21.15.27:0/1897938128 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd31c0cc910 con 0x7fd31c0b2190 2023-10-06T01:42:24.426 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.423+0000 7fd321da3700 1 --2- 172.21.15.27:0/1897938128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd31c0b2190 0x7fd31c0b25d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fd30c009e60 tx=0x7fd30c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=969a1116df4a1bac server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:24.426 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.423+0000 7fd31bfff700 1 -- 172.21.15.27:0/1897938128 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30c00e070 con 0x7fd31c0b2190 2023-10-06T01:42:24.426 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.423+0000 7fd31bfff700 1 -- 172.21.15.27:0/1897938128 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd30c00bc30 con 0x7fd31c0b2190 2023-10-06T01:42:24.426 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.423+0000 7fd31bfff700 1 -- 172.21.15.27:0/1897938128 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30c012430 con 0x7fd31c0b2190 2023-10-06T01:42:24.427 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.423+0000 7fd3225a4700 1 -- 172.21.15.27:0/1897938128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd31c0b2190 msgr2=0x7fd31c0b25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:24.427 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.423+0000 7fd3225a4700 1 --2- 172.21.15.27:0/1897938128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd31c0b2190 0x7fd31c0b25d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fd30c009e60 tx=0x7fd30c009780 comp rx=0 tx=0).stop 2023-10-06T01:42:24.427 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.423+0000 7fd3225a4700 1 -- 172.21.15.27:0/1897938128 shutdown_connections 2023-10-06T01:42:24.427 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.423+0000 7fd3225a4700 1 --2- 172.21.15.27:0/1897938128 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd31c0b2190 0x7fd31c0b25d0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:24.427 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.423+0000 7fd3225a4700 1 --2- 172.21.15.27:0/1897938128 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd31c157b70 0x7fd31c158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:24.427 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.423+0000 7fd3225a4700 1 -- 172.21.15.27:0/1897938128 >> 172.21.15.27:0/1897938128 conn(0x7fd31c06d5f0 msgr2=0x7fd31c06d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:24.427 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.423+0000 7fd3225a4700 1 -- 172.21.15.27:0/1897938128 shutdown_connections 2023-10-06T01:42:24.427 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.423+0000 7fd3225a4700 1 -- 172.21.15.27:0/1897938128 wait complete. 2023-10-06T01:42:24.430 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.424+0000 7fd3225a4700 1 Processor -- start 2023-10-06T01:42:24.430 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.425+0000 7fd3225a4700 1 -- start start 2023-10-06T01:42:24.430 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.425+0000 7fd3225a4700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd31c0b2190 0x7fd31c15b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:24.430 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.425+0000 7fd3225a4700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd31c157b70 0x7fd31c068fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:24.430 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.425+0000 7fd3225a4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd31c159210 0x7fd31c069510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:24.430 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.425+0000 7fd3225a4700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd31c156aa0 con 0x7fd31c159210 2023-10-06T01:42:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.425+0000 7fd3225a4700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd31c1561b0 con 0x7fd31c0b2190 2023-10-06T01:42:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.425+0000 7fd3225a4700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd31c156450 con 0x7fd31c157b70 2023-10-06T01:42:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.426+0000 7fd321da3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd31c159210 0x7fd31c069510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.426+0000 7fd321da3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd31c159210 0x7fd31c069510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60170/0 (socket says 172.21.15.27:60170) 2023-10-06T01:42:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.426+0000 7fd321da3700 1 -- 172.21.15.27:0/386646449 learned_addr learned my addr 172.21.15.27:0/386646449 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.426+0000 7fd3215a2700 1 --2- 172.21.15.27:0/386646449 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd31c0b2190 0x7fd31c15b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.426+0000 7fd320da1700 1 --2- 172.21.15.27:0/386646449 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd31c157b70 0x7fd31c068fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.426+0000 7fd321da3700 1 -- 172.21.15.27:0/386646449 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd31c157b70 msgr2=0x7fd31c068fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.426+0000 7fd321da3700 1 --2- 172.21.15.27:0/386646449 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd31c157b70 0x7fd31c068fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.426+0000 7fd321da3700 1 -- 172.21.15.27:0/386646449 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd31c0b2190 msgr2=0x7fd31c15b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.426+0000 7fd321da3700 1 --2- 172.21.15.27:0/386646449 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd31c0b2190 0x7fd31c15b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:24.431 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.426+0000 7fd321da3700 1 -- 172.21.15.27:0/386646449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd30c009a20 con 0x7fd31c159210 2023-10-06T01:42:24.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.426+0000 7fd321da3700 1 --2- 172.21.15.27:0/386646449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd31c159210 0x7fd31c069510 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd30c013000 tx=0x7fd30c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:24.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.426+0000 7fd3197fa700 1 -- 172.21.15.27:0/386646449 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30c012e50 con 0x7fd31c159210 2023-10-06T01:42:24.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.426+0000 7fd3197fa700 1 -- 172.21.15.27:0/386646449 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd30c01d630 con 0x7fd31c159210 2023-10-06T01:42:24.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.426+0000 7fd3225a4700 1 -- 172.21.15.27:0/386646449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd31c156730 con 0x7fd31c159210 2023-10-06T01:42:24.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.426+0000 7fd3197fa700 1 -- 172.21.15.27:0/386646449 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30c01da10 con 0x7fd31c159210 2023-10-06T01:42:24.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.426+0000 7fd3225a4700 1 -- 172.21.15.27:0/386646449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd31c06adc0 con 0x7fd31c159210 2023-10-06T01:42:24.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.427+0000 7fd3225a4700 1 -- 172.21.15.27:0/386646449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd31c156c10 con 0x7fd31c159210 2023-10-06T01:42:24.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.428+0000 7fd3197fa700 1 -- 172.21.15.27:0/386646449 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fd30c024070 con 0x7fd31c159210 2023-10-06T01:42:24.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.428+0000 7fd3197fa700 1 --2- 172.21.15.27:0/386646449 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd308066f30 0x7fd3080693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:24.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.428+0000 7fd3197fa700 1 -- 172.21.15.27:0/386646449 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6453+0+0 (secure 0 0 0) 0x7fd30c0870c0 con 0x7fd31c159210 2023-10-06T01:42:24.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.428+0000 7fd3215a2700 1 --2- 172.21.15.27:0/386646449 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd308066f30 0x7fd3080693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:24.432 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.429+0000 7fd3215a2700 1 --2- 172.21.15.27:0/386646449 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd308066f30 0x7fd3080693e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd314006fd0 tx=0x7fd314008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:24.434 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.433+0000 7fd3197fa700 1 -- 172.21.15.27:0/386646449 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd30c0540f0 con 0x7fd31c159210 2023-10-06T01:42:24.630 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.628+0000 7fd3225a4700 1 -- 172.21.15.27:0/386646449 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd31c15be00 con 0x7fd31c159210 2023-10-06T01:42:24.630 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.629+0000 7fd3197fa700 1 -- 172.21.15.27:0/386646449 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+18425 (secure 0 0 0) 0x7fd30c057710 con 0x7fd31c159210 2023-10-06T01:42:24.631 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:24.631 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":34,"fsid":"20a7bff0-74f8-4f2f-b6c6-899e804bc965","created":"2023-10-06T01:40:54.474215+0000","modified":"2023-10-06T01:42:18.365024+0000","last_up_change":"2023-10-06T01:41:05.242847+0000","last_in_change":"2023-10-06T01:40:56.890960+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-10-06T01:41:05.685386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-10-06T01:41:06.092995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-10-06T01:42:01.295053+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-10-06T01:42:02.005152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":7,"pool_name":"data_new","create_time":"2023-10-06T01:42:16.105156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b902c86d-800c-40c5-b152-b545e50d02c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":90644},{"type":"v1","addr":"172.21.15.27:6801","nonce":90644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":90644},{"type":"v1","addr":"172.21.15.27:6803","nonce":90644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":90644},{"type":"v1","addr":"172.21.15.27:6807","nonce":90644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":90644},{"type":"v1","addr":"172.21.15.27:6805","nonce":90644}]},"public_addr":"172.21.15.27:6801/90644","cluster_addr":"172.21.15.27:6803/90644","heartbeat_back_addr":"172.21.15.27:6807/90644","heartbeat_front_addr":"172.21.15.27:6805/90644","state":["exists","up"]},{"osd":1,"uuid":"5b4a70e5-b06e-48cd-bfc4-7962c5d05386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":90643},{"type":"v1","addr":"172.21.15.27:6809","nonce":90643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":90643},{"type":"v1","addr":"172.21.15.27:6811","nonce":90643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":90643},{"type":"v1","addr":"172.21.15.27:6815","nonce":90643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":90643},{"type":"v1","addr":"172.21.15.27:6813","nonce":90643}]},"public_addr":"172.21.15.27:6809/90643","cluster_addr":"172.21.15.27:6811/90643","heartbeat_back_addr":"172.21.15.27:6815/90643","heartbeat_front_addr":"172.21.15.27:6813/90643","state":["exists","up"]},{"osd":2,"uuid":"22d9de46-4073-4336-9122-5227e601eca0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":90645},{"type":"v1","addr":"172.21.15.27:6817","nonce":90645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":90645},{"type":"v1","addr":"172.21.15.27:6819","nonce":90645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":90645},{"type":"v1","addr":"172.21.15.27:6823","nonce":90645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":90645},{"type":"v1","addr":"172.21.15.27:6821","nonce":90645}]},"public_addr":"172.21.15.27:6817/90645","cluster_addr":"172.21.15.27:6819/90645","heartbeat_back_addr":"172.21.15.27:6823/90645","heartbeat_front_addr":"172.21.15.27:6821/90645","state":["exists","up"]},{"osd":3,"uuid":"438bb2d3-34e2-4b7c-9734-8360c1b67650","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":90668},{"type":"v1","addr":"172.21.15.27:6825","nonce":90668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":90668},{"type":"v1","addr":"172.21.15.27:6827","nonce":90668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":90668},{"type":"v1","addr":"172.21.15.27:6831","nonce":90668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":90668},{"type":"v1","addr":"172.21.15.27:6829","nonce":90668}]},"public_addr":"172.21.15.27:6825/90668","cluster_addr":"172.21.15.27:6827/90668","heartbeat_back_addr":"172.21.15.27:6831/90668","heartbeat_front_addr":"172.21.15.27:6829/90668","state":["exists","up"]},{"osd":4,"uuid":"e1956a8e-e0f8-4e7c-acbb-40037415a354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":90863},{"type":"v1","addr":"172.21.15.124:6801","nonce":90863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":90863},{"type":"v1","addr":"172.21.15.124:6803","nonce":90863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":90863},{"type":"v1","addr":"172.21.15.124:6807","nonce":90863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":90863},{"type":"v1","addr":"172.21.15.124:6805","nonce":90863}]},"public_addr":"172.21.15.124:6801/90863","cluster_addr":"172.21.15.124:6803/90863","heartbeat_back_addr":"172.21.15.124:6807/90863","heartbeat_front_addr":"172.21.15.124:6805/90863","state":["exists","up"]},{"osd":5,"uuid":"306c59d4-846d-4b73-b1c9-051bec9e9368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":90885},{"type":"v1","addr":"172.21.15.124:6817","nonce":90885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":90885},{"type":"v1","addr":"172.21.15.124:6819","nonce":90885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":90885},{"type":"v1","addr":"172.21.15.124:6823","nonce":90885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":90885},{"type":"v1","addr":"172.21.15.124:6821","nonce":90885}]},"public_addr":"172.21.15.124:6817/90885","cluster_addr":"172.21.15.124:6819/90885","heartbeat_back_addr":"172.21.15.124:6823/90885","heartbeat_front_addr":"172.21.15.124:6821/90885","state":["exists","up"]},{"osd":6,"uuid":"15463e53-ac8d-47d1-b4f6-c0d31f29e4a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":90864},{"type":"v1","addr":"172.21.15.124:6809","nonce":90864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":90864},{"type":"v1","addr":"172.21.15.124:6811","nonce":90864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":90864},{"type":"v1","addr":"172.21.15.124:6815","nonce":90864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":90864},{"type":"v1","addr":"172.21.15.124:6813","nonce":90864}]},"public_addr":"172.21.15.124:6809/90864","cluster_addr":"172.21.15.124:6811/90864","heartbeat_back_addr":"172.21.15.124:6815/90864","heartbeat_front_addr":"172.21.15.124:6813/90864","state":["exists","up"]},{"osd":7,"uuid":"ae463a90-a9d3-498f-86fa-47a35b3fe889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":90894},{"type":"v1","addr":"172.21.15.124:6825","nonce":90894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":90894},{"type":"v1","addr":"172.21.15.124:6827","nonce":90894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":90894},{"type":"v1","addr":"172.21.15.124:6831","nonce":90894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":90894},{"type":"v1","addr":"172.21.15.124:6829","nonce":90894}]},"public_addr":"172.21.15.124:6825/90894","cluster_addr":"172.21.15.124:6827/90894","heartbeat_back_addr":"172.21.15.124:6831/90894","heartbeat_front_addr":"172.21.15.124:6829/90894","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.827704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.839951+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.032230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.050605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.987908+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.031290+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.080707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.053692+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T01:42:24.633 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.631+0000 7fd3225a4700 1 -- 172.21.15.27:0/386646449 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd308066f30 msgr2=0x7fd3080693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:24.633 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.631+0000 7fd3225a4700 1 --2- 172.21.15.27:0/386646449 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd308066f30 0x7fd3080693e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd314006fd0 tx=0x7fd314008040 comp rx=0 tx=0).stop 2023-10-06T01:42:24.633 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.631+0000 7fd3225a4700 1 -- 172.21.15.27:0/386646449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd31c159210 msgr2=0x7fd31c069510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:24.633 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.631+0000 7fd3225a4700 1 --2- 172.21.15.27:0/386646449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd31c159210 0x7fd31c069510 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd30c013000 tx=0x7fd30c019000 comp rx=0 tx=0).stop 2023-10-06T01:42:24.633 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.631+0000 7fd3225a4700 1 -- 172.21.15.27:0/386646449 shutdown_connections 2023-10-06T01:42:24.633 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.631+0000 7fd3225a4700 1 --2- 172.21.15.27:0/386646449 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fd308066f30 0x7fd3080693e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:24.633 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.631+0000 7fd3225a4700 1 --2- 172.21.15.27:0/386646449 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fd31c159210 0x7fd31c069510 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:24.633 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.631+0000 7fd3225a4700 1 --2- 172.21.15.27:0/386646449 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fd31c0b2190 0x7fd31c15b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:24.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.631+0000 7fd3225a4700 1 --2- 172.21.15.27:0/386646449 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fd31c157b70 0x7fd31c068fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:24.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.631+0000 7fd3225a4700 1 -- 172.21.15.27:0/386646449 >> 172.21.15.27:0/386646449 conn(0x7fd31c06d5f0 msgr2=0x7fd31c150ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:24.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.631+0000 7fd3225a4700 1 -- 172.21.15.27:0/386646449 shutdown_connections 2023-10-06T01:42:24.634 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.631+0000 7fd3225a4700 1 -- 172.21.15.27:0/386646449 wait complete. 2023-10-06T01:42:24.645 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-06T01:42:24.645 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T01:42:24.769 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.762+0000 7f2f18f57700 1 Processor -- start 2023-10-06T01:42:24.769 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.762+0000 7f2f18f57700 1 -- start start 2023-10-06T01:42:24.769 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.763+0000 7f2f18f57700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2f1415d560 0x7f2f140d0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:24.770 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.763+0000 7f2f18f57700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2f1415d930 0x7f2f140b1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:24.770 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.763+0000 7f2f18f57700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f140b2ad0 con 0x7f2f141591b0 2023-10-06T01:42:24.770 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.763+0000 7f2f18f57700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f140b4160 con 0x7f2f1415d930 2023-10-06T01:42:24.770 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.763+0000 7f2f18f57700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f140b1f80 con 0x7f2f1415d560 2023-10-06T01:42:24.770 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.764+0000 7f2f11d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2f1415d930 0x7f2f140b1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:24.771 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.764+0000 7f2f12d9d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f2f141591b0 0x7f2f14159630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:40482/0 (socket says 172.21.15.27:40482) 2023-10-06T01:42:24.771 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.764+0000 7f2f11d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2f1415d930 0x7f2f140b1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60186/0 (socket says 172.21.15.27:60186) 2023-10-06T01:42:24.771 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.764+0000 7f2f12d9d700 1 -- 172.21.15.27:0/4292085429 learned_addr learned my addr 172.21.15.27:0/4292085429 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:24.771 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.764+0000 7f2f1259c700 1 --2- 172.21.15.27:0/4292085429 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2f1415d560 0x7f2f140d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:24.771 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.764+0000 7f2f11d9b700 1 -- 172.21.15.27:0/4292085429 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2f1415d560 msgr2=0x7f2f140d0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:24.771 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.764+0000 7f2f11d9b700 1 --2- 172.21.15.27:0/4292085429 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2f1415d560 0x7f2f140d0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:24.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.764+0000 7f2f11d9b700 1 -- 172.21.15.27:0/4292085429 >> v1:172.21.15.27:6789/0 conn(0x7f2f141591b0 legacy=0x7f2f14159630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:24.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.764+0000 7f2f11d9b700 1 -- 172.21.15.27:0/4292085429 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f140b2230 con 0x7f2f1415d930 2023-10-06T01:42:24.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.764+0000 7f2f1159a700 1 -- 172.21.15.27:0/4292085429 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4005323498 0 0) 0x7f2f140b2ad0 con 0x7f2f141591b0 2023-10-06T01:42:24.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.764+0000 7f2f1259c700 1 --2- 172.21.15.27:0/4292085429 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2f1415d560 0x7f2f140d0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:42:24.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.764+0000 7f2f11d9b700 1 --2- 172.21.15.27:0/4292085429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f1415d930 0x7f2f140b1a40 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f2efc0098e0 tx=0x7f2efc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9814cbfe9a9ceaf server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:24.772 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.764+0000 7f2f1159a700 1 -- 172.21.15.27:0/4292085429 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2efc009e80 con 0x7f2f1415d930 2023-10-06T01:42:24.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.764+0000 7f2f1159a700 1 -- 172.21.15.27:0/4292085429 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2efc00b860 con 0x7f2f1415d930 2023-10-06T01:42:24.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.764+0000 7f2f1159a700 1 -- 172.21.15.27:0/4292085429 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2efc00bbe0 con 0x7f2f1415d930 2023-10-06T01:42:24.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.765+0000 7f2f18f57700 1 -- 172.21.15.27:0/4292085429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f1415d930 msgr2=0x7f2f140b1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:24.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.765+0000 7f2f18f57700 1 --2- 172.21.15.27:0/4292085429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f1415d930 0x7f2f140b1a40 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f2efc0098e0 tx=0x7f2efc0091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:24.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.765+0000 7f2f18f57700 1 -- 172.21.15.27:0/4292085429 shutdown_connections 2023-10-06T01:42:24.773 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.765+0000 7f2f18f57700 1 --2- 172.21.15.27:0/4292085429 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2f1415d560 0x7f2f140d0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:24.774 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.765+0000 7f2f18f57700 1 --2- 172.21.15.27:0/4292085429 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f1415d930 0x7f2f140b1a40 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:24.774 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.765+0000 7f2f18f57700 1 -- 172.21.15.27:0/4292085429 >> 172.21.15.27:0/4292085429 conn(0x7f2f140d1180 msgr2=0x7f2f14069640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:24.774 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.765+0000 7f2f18f57700 1 -- 172.21.15.27:0/4292085429 shutdown_connections 2023-10-06T01:42:24.774 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.765+0000 7f2f18f57700 1 -- 172.21.15.27:0/4292085429 wait complete. 2023-10-06T01:42:24.774 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.766+0000 7f2f18f57700 1 Processor -- start 2023-10-06T01:42:24.774 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.766+0000 7f2f18f57700 1 -- start start 2023-10-06T01:42:24.775 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.767+0000 7f2f18f57700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2f141591b0 0x7f2f1415c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:24.775 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.767+0000 7f2f18f57700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2f1415d560 0x7f2f141441a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:24.775 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.767+0000 7f2f18f57700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f1415d930 0x7f2f1415adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:24.775 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.767+0000 7f2f18f57700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f141446e0 con 0x7f2f1415d930 2023-10-06T01:42:24.775 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.767+0000 7f2f18f57700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1415c400 con 0x7f2f1415d560 2023-10-06T01:42:24.775 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.767+0000 7f2f18f57700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1415b7f0 con 0x7f2f141591b0 2023-10-06T01:42:24.775 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.767+0000 7f2f12d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f1415d930 0x7f2f1415adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:24.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.767+0000 7f2f12d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f1415d930 0x7f2f1415adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60192/0 (socket says 172.21.15.27:60192) 2023-10-06T01:42:24.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.767+0000 7f2f12d9d700 1 -- 172.21.15.27:0/2483551681 learned_addr learned my addr 172.21.15.27:0/2483551681 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:24.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.767+0000 7f2f11d9b700 1 --2- 172.21.15.27:0/2483551681 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2f1415d560 0x7f2f141441a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:24.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.767+0000 7f2f1259c700 1 --2- 172.21.15.27:0/2483551681 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2f141591b0 0x7f2f1415c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:24.776 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.767+0000 7f2f12d9d700 1 -- 172.21.15.27:0/2483551681 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2f141591b0 msgr2=0x7f2f1415c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:24.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.767+0000 7f2f12d9d700 1 --2- 172.21.15.27:0/2483551681 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2f141591b0 0x7f2f1415c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:24.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.767+0000 7f2f12d9d700 1 -- 172.21.15.27:0/2483551681 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2f1415d560 msgr2=0x7f2f141441a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:24.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.767+0000 7f2f12d9d700 1 --2- 172.21.15.27:0/2483551681 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2f1415d560 0x7f2f141441a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:24.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.767+0000 7f2f12d9d700 1 -- 172.21.15.27:0/2483551681 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2efc009420 con 0x7f2f1415d930 2023-10-06T01:42:24.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.767+0000 7f2f12d9d700 1 --2- 172.21.15.27:0/2483551681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f1415d930 0x7f2f1415adc0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f2f0400c7e0 tx=0x7f2f04007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:24.777 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.767+0000 7f2f0a7fc700 1 -- 172.21.15.27:0/2483551681 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f04005c00 con 0x7f2f1415d930 2023-10-06T01:42:24.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.768+0000 7f2f0a7fc700 1 -- 172.21.15.27:0/2483551681 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f04005d80 con 0x7f2f1415d930 2023-10-06T01:42:24.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.768+0000 7f2f18f57700 1 -- 172.21.15.27:0/2483551681 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f1415bad0 con 0x7f2f1415d930 2023-10-06T01:42:24.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.768+0000 7f2f0a7fc700 1 -- 172.21.15.27:0/2483551681 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f04007320 con 0x7f2f1415d930 2023-10-06T01:42:24.778 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.768+0000 7f2f18f57700 1 -- 172.21.15.27:0/2483551681 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f1415c090 con 0x7f2f1415d930 2023-10-06T01:42:24.779 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.768+0000 7f2f18f57700 1 -- 172.21.15.27:0/2483551681 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f14064050 con 0x7f2f1415d930 2023-10-06T01:42:24.779 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.769+0000 7f2f0a7fc700 1 -- 172.21.15.27:0/2483551681 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2f0401b3b0 con 0x7f2f1415d930 2023-10-06T01:42:24.780 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.769+0000 7f2f0a7fc700 1 --2- 172.21.15.27:0/2483551681 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2ef4066f30 0x7f2ef40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:24.780 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.769+0000 7f2f0a7fc700 1 -- 172.21.15.27:0/2483551681 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f2f04083c60 con 0x7f2f1415d930 2023-10-06T01:42:24.780 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.769+0000 7f2f1259c700 1 --2- 172.21.15.27:0/2483551681 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2ef4066f30 0x7f2ef40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:24.780 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.770+0000 7f2f1259c700 1 --2- 172.21.15.27:0/2483551681 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2ef4066f30 0x7f2ef40693e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c007740 tx=0x7f2f0c0099d0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:24.780 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:24.773+0000 7f2f0a7fc700 1 -- 172.21.15.27:0/2483551681 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2f04050be0 con 0x7f2f1415d930 2023-10-06T01:42:25.009 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.008+0000 7f2f18f57700 1 -- 172.21.15.27:0/2483551681 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f140b2510 con 0x7f2f1415d930 2023-10-06T01:42:25.010 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.009+0000 7f2f0a7fc700 1 -- 172.21.15.27:0/2483551681 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4599 (secure 0 0 0) 0x7f2f04017070 con 0x7f2f1415d930 2023-10-06T01:42:25.010 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:25.011 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4870,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6836/1043044736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":1043044736},{"type":"v1","addr":"172.21.15.124:6836","nonce":1043044736}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4962,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/1207987117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1207987117},{"type":"v1","addr":"172.21.15.27:6834","nonce":1207987117}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4965,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/943261840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":943261840},{"type":"v1","addr":"172.21.15.27:6835","nonce":943261840}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T01:42:03.036784+0000","modified":"2023-10-06T01:42:17.363571+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4974},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4974":{"gid":4974,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.124:6837/325247112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6835","nonce":325247112},{"type":"v1","addr":"172.21.15.124:6837","nonce":325247112}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6,7],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-10-06T01:42:25.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.011+0000 7f2f18f57700 1 -- 172.21.15.27:0/2483551681 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2ef4066f30 msgr2=0x7f2ef40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:25.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.011+0000 7f2f18f57700 1 --2- 172.21.15.27:0/2483551681 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2ef4066f30 0x7f2ef40693e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2f0c007740 tx=0x7f2f0c0099d0 comp rx=0 tx=0).stop 2023-10-06T01:42:25.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.011+0000 7f2f18f57700 1 -- 172.21.15.27:0/2483551681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f1415d930 msgr2=0x7f2f1415adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:25.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.011+0000 7f2f18f57700 1 --2- 172.21.15.27:0/2483551681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f1415d930 0x7f2f1415adc0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f2f0400c7e0 tx=0x7f2f04007d90 comp rx=0 tx=0).stop 2023-10-06T01:42:25.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.011+0000 7f2f18f57700 1 -- 172.21.15.27:0/2483551681 shutdown_connections 2023-10-06T01:42:25.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.011+0000 7f2f18f57700 1 --2- 172.21.15.27:0/2483551681 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2ef4066f30 0x7f2ef40693e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:25.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.011+0000 7f2f18f57700 1 --2- 172.21.15.27:0/2483551681 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2f141591b0 0x7f2f1415c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:25.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.011+0000 7f2f18f57700 1 --2- 172.21.15.27:0/2483551681 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2f1415d930 0x7f2f1415adc0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:25.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.011+0000 7f2f18f57700 1 --2- 172.21.15.27:0/2483551681 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2f1415d560 0x7f2f141441a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:25.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.011+0000 7f2f18f57700 1 -- 172.21.15.27:0/2483551681 >> 172.21.15.27:0/2483551681 conn(0x7f2f140d1180 msgr2=0x7f2f140b5ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:25.014 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.011+0000 7f2f18f57700 1 -- 172.21.15.27:0/2483551681 shutdown_connections 2023-10-06T01:42:25.014 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.011+0000 7f2f18f57700 1 -- 172.21.15.27:0/2483551681 wait complete. 2023-10-06T01:42:25.014 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 16 2023-10-06T01:42:25.024 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T01:42:25.142 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.141+0000 7f5638b1b700 1 Processor -- start 2023-10-06T01:42:25.142 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.141+0000 7f5638b1b700 1 -- start start 2023-10-06T01:42:25.144 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.142+0000 7f5638b1b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f56340cbe50 0x7f56340c66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:25.145 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.142+0000 7f5638b1b700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f56340c6bf0 0x7f56340ae5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:25.145 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.142+0000 7f5638b1b700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56340ae140 con 0x7f56340cdc00 2023-10-06T01:42:25.145 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.142+0000 7f5638b1b700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56340c7020 con 0x7f56340cbe50 2023-10-06T01:42:25.145 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.142+0000 7f5638b1b700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56340cc2a0 con 0x7f56340c6bf0 2023-10-06T01:42:25.145 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.142+0000 7f5631d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f56340cbe50 0x7f56340c66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:25.146 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.142+0000 7f5631d9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f56340cbe50 0x7f56340c66b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60204/0 (socket says 172.21.15.27:60204) 2023-10-06T01:42:25.146 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.142+0000 7f5631d9b700 1 -- 172.21.15.27:0/2939871127 learned_addr learned my addr 172.21.15.27:0/2939871127 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:25.146 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.142+0000 7f5632d9d700 1 --2- 172.21.15.27:0/2939871127 >> v2:172.21.15.124:3300/0 conn(0x7f56340c6bf0 0x7f56340ae5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:25.146 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.142+0000 7f5631d9b700 1 -- 172.21.15.27:0/2939871127 >> v2:172.21.15.124:3300/0 conn(0x7f56340c6bf0 msgr2=0x7f56340ae5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:25.146 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.142+0000 7f5631d9b700 1 --2- 172.21.15.27:0/2939871127 >> v2:172.21.15.124:3300/0 conn(0x7f56340c6bf0 0x7f56340ae5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:25.147 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.142+0000 7f5631d9b700 1 -- 172.21.15.27:0/2939871127 >> v1:172.21.15.124:6789/0 conn(0x7f56340cdc00 legacy=0x7f56340d1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:25.147 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.142+0000 7f5631d9b700 1 -- 172.21.15.27:0/2939871127 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56340ad530 con 0x7f56340cbe50 2023-10-06T01:42:25.147 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.142+0000 7f5631d9b700 1 --2- 172.21.15.27:0/2939871127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56340cbe50 0x7f56340c66b0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f563408a900 tx=0x7f5628009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=851a7222820ef98c server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:25.147 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.143+0000 7f563159a700 1 -- 172.21.15.27:0/2939871127 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562800b820 con 0x7f56340cbe50 2023-10-06T01:42:25.147 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.143+0000 7f563159a700 1 -- 172.21.15.27:0/2939871127 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f562800b980 con 0x7f56340cbe50 2023-10-06T01:42:25.147 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.143+0000 7f563159a700 1 -- 172.21.15.27:0/2939871127 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562800bd20 con 0x7f56340cbe50 2023-10-06T01:42:25.148 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.143+0000 7f5638b1b700 1 -- 172.21.15.27:0/2939871127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56340cbe50 msgr2=0x7f56340c66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:25.148 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.143+0000 7f5638b1b700 1 --2- 172.21.15.27:0/2939871127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56340cbe50 0x7f56340c66b0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f563408a900 tx=0x7f5628009250 comp rx=0 tx=0).stop 2023-10-06T01:42:25.148 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.143+0000 7f5638b1b700 1 -- 172.21.15.27:0/2939871127 shutdown_connections 2023-10-06T01:42:25.148 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.143+0000 7f5638b1b700 1 --2- 172.21.15.27:0/2939871127 >> v2:172.21.15.124:3300/0 conn(0x7f56340c6bf0 0x7f56340ae5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:25.148 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.143+0000 7f5638b1b700 1 --2- 172.21.15.27:0/2939871127 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56340cbe50 0x7f56340c66b0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:25.149 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.143+0000 7f5638b1b700 1 -- 172.21.15.27:0/2939871127 >> 172.21.15.27:0/2939871127 conn(0x7f5634099990 msgr2=0x7f56340b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:25.149 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.143+0000 7f5638b1b700 1 -- 172.21.15.27:0/2939871127 shutdown_connections 2023-10-06T01:42:25.149 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.143+0000 7f5638b1b700 1 -- 172.21.15.27:0/2939871127 wait complete. 2023-10-06T01:42:25.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.144+0000 7f5638b1b700 1 Processor -- start 2023-10-06T01:42:25.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.144+0000 7f5638b1b700 1 -- start start 2023-10-06T01:42:25.151 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.145+0000 7f5638b1b700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f56340c6bf0 0x7f56341582f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:25.152 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.145+0000 7f5638b1b700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f56340cbe50 0x7f56341696e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:25.152 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.145+0000 7f5638b1b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56340cdc00 0x7f56340697e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:25.152 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.145+0000 7f5638b1b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56340c8270 con 0x7f56340cdc00 2023-10-06T01:42:25.152 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.145+0000 7f5638b1b700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56340c7940 con 0x7f56340cbe50 2023-10-06T01:42:25.152 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.145+0000 7f5638b1b700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56340c7c20 con 0x7f56340c6bf0 2023-10-06T01:42:25.152 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.145+0000 7f5632d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56340cdc00 0x7f56340697e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:25.152 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.145+0000 7f5632d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56340cdc00 0x7f56340697e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:60220/0 (socket says 172.21.15.27:60220) 2023-10-06T01:42:25.152 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.145+0000 7f5632d9d700 1 -- 172.21.15.27:0/3950238603 learned_addr learned my addr 172.21.15.27:0/3950238603 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:25.152 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.145+0000 7f5631d9b700 1 --2- 172.21.15.27:0/3950238603 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f56340cbe50 0x7f56341696e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:25.152 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.145+0000 7f563259c700 1 --2- 172.21.15.27:0/3950238603 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f56340c6bf0 0x7f56341582f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:25.153 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.145+0000 7f5632d9d700 1 -- 172.21.15.27:0/3950238603 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f56340c6bf0 msgr2=0x7f56341582f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:25.153 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.145+0000 7f5632d9d700 1 --2- 172.21.15.27:0/3950238603 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f56340c6bf0 0x7f56341582f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:25.153 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.145+0000 7f5632d9d700 1 -- 172.21.15.27:0/3950238603 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f56340cbe50 msgr2=0x7f56341696e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:25.153 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.145+0000 7f5632d9d700 1 --2- 172.21.15.27:0/3950238603 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f56340cbe50 0x7f56341696e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:25.153 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.145+0000 7f5632d9d700 1 -- 172.21.15.27:0/3950238603 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56280094f0 con 0x7f56340cdc00 2023-10-06T01:42:25.153 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.146+0000 7f5632d9d700 1 --2- 172.21.15.27:0/3950238603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56340cdc00 0x7f56340697e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f562c00ceb0 tx=0x7f562c007870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:25.153 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.146+0000 7f56227fc700 1 -- 172.21.15.27:0/3950238603 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562c00ad90 con 0x7f56340cdc00 2023-10-06T01:42:25.153 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.146+0000 7f56227fc700 1 -- 172.21.15.27:0/3950238603 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f562c016430 con 0x7f56340cdc00 2023-10-06T01:42:25.153 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.146+0000 7f56227fc700 1 -- 172.21.15.27:0/3950238603 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562c0167d0 con 0x7f56340cdc00 2023-10-06T01:42:25.153 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.146+0000 7f5638b1b700 1 -- 172.21.15.27:0/3950238603 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56340c7f00 con 0x7f56340cdc00 2023-10-06T01:42:25.153 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.146+0000 7f5638b1b700 1 -- 172.21.15.27:0/3950238603 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5634169c20 con 0x7f56340cdc00 2023-10-06T01:42:25.154 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.147+0000 7f5638b1b700 1 -- 172.21.15.27:0/3950238603 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56340c8420 con 0x7f56340cdc00 2023-10-06T01:42:25.154 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.147+0000 7f56227fc700 1 -- 172.21.15.27:0/3950238603 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f562c003020 con 0x7f56340cdc00 2023-10-06T01:42:25.154 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.148+0000 7f56227fc700 1 --2- 172.21.15.27:0/3950238603 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5614066f30 0x7f56140693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:25.154 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.148+0000 7f56227fc700 1 -- 172.21.15.27:0/3950238603 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f562c012070 con 0x7f56340cdc00 2023-10-06T01:42:25.154 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.148+0000 7f563259c700 1 --2- 172.21.15.27:0/3950238603 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5614066f30 0x7f56140693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:25.154 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.148+0000 7f563259c700 1 --2- 172.21.15.27:0/3950238603 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5614066f30 0x7f56140693e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5624006fd0 tx=0x7f5624009060 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:25.154 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.152+0000 7f56227fc700 1 -- 172.21.15.27:0/3950238603 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f56340c8420 con 0x7f56340cdc00 2023-10-06T01:42:25.390 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.389+0000 7f5638b1b700 1 -- 172.21.15.27:0/3950238603 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5634169ea0 con 0x7f56340cdc00 2023-10-06T01:42:25.391 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.390+0000 7f56227fc700 1 -- 172.21.15.27:0/3950238603 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4599 (secure 0 0 0) 0x7f562c052540 con 0x7f56340cdc00 2023-10-06T01:42:25.391 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:25.391 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4870,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6836/1043044736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":1043044736},{"type":"v1","addr":"172.21.15.124:6836","nonce":1043044736}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4962,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/1207987117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1207987117},{"type":"v1","addr":"172.21.15.27:6834","nonce":1207987117}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4965,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/943261840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":943261840},{"type":"v1","addr":"172.21.15.27:6835","nonce":943261840}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-06T01:42:03.036784+0000","modified":"2023-10-06T01:42:17.363571+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4974},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4974":{"gid":4974,"name":"d","rank":0,"incarnation":12,"state":"up:active","state_seq":3,"addr":"172.21.15.124:6837/325247112","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6835","nonce":325247112},{"type":"v1","addr":"172.21.15.124:6837","nonce":325247112}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6,7],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2023-10-06T01:42:25.394 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.392+0000 7f5638b1b700 1 -- 172.21.15.27:0/3950238603 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5614066f30 msgr2=0x7f56140693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:25.394 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.392+0000 7f5638b1b700 1 --2- 172.21.15.27:0/3950238603 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5614066f30 0x7f56140693e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f5624006fd0 tx=0x7f5624009060 comp rx=0 tx=0).stop 2023-10-06T01:42:25.394 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.392+0000 7f5638b1b700 1 -- 172.21.15.27:0/3950238603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56340cdc00 msgr2=0x7f56340697e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:25.394 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.392+0000 7f5638b1b700 1 --2- 172.21.15.27:0/3950238603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56340cdc00 0x7f56340697e0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f562c00ceb0 tx=0x7f562c007870 comp rx=0 tx=0).stop 2023-10-06T01:42:25.394 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.392+0000 7f5638b1b700 1 -- 172.21.15.27:0/3950238603 shutdown_connections 2023-10-06T01:42:25.394 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.392+0000 7f5638b1b700 1 --2- 172.21.15.27:0/3950238603 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f5614066f30 0x7f56140693e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:25.394 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.392+0000 7f5638b1b700 1 --2- 172.21.15.27:0/3950238603 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f56340c6bf0 0x7f56341582f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:25.394 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.392+0000 7f5638b1b700 1 --2- 172.21.15.27:0/3950238603 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f56340cdc00 0x7f56340697e0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:25.394 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.392+0000 7f5638b1b700 1 --2- 172.21.15.27:0/3950238603 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f56340cbe50 0x7f56341696e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:25.395 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.392+0000 7f5638b1b700 1 -- 172.21.15.27:0/3950238603 >> 172.21.15.27:0/3950238603 conn(0x7f5634099990 msgr2=0x7f56340b1920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:25.395 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.392+0000 7f5638b1b700 1 -- 172.21.15.27:0/3950238603 shutdown_connections 2023-10-06T01:42:25.395 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.392+0000 7f5638b1b700 1 -- 172.21.15.27:0/3950238603 wait complete. 2023-10-06T01:42:25.395 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 16 2023-10-06T01:42:25.405 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T01:42:25.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.522+0000 7f6bdab57700 1 Processor -- start 2023-10-06T01:42:25.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.522+0000 7f6bdab57700 1 -- start start 2023-10-06T01:42:25.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.523+0000 7f6bdab57700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6bd40cdc00 0x7f6bd40cbe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:25.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.523+0000 7f6bdab57700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f6bd40d1030 0x7f6bd40c66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:25.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.523+0000 7f6bdab57700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6bd40c6bf0 0x7f6bd40ae5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:25.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.523+0000 7f6bdab57700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bd40cc390 con 0x7f6bd40c6bf0 2023-10-06T01:42:25.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.523+0000 7f6bdab57700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bd40c7070 con 0x7f6bd40d1030 2023-10-06T01:42:25.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.523+0000 7f6bdab57700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd40ae140 con 0x7f6bd40cdc00 2023-10-06T01:42:25.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.523+0000 7f6bd90f4700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6bd40c6bf0 0x7f6bd40ae5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:25.527 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.523+0000 7f6bd90f4700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6bd40c6bf0 0x7f6bd40ae5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48218/0 (socket says 172.21.15.27:48218) 2023-10-06T01:42:25.527 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.523+0000 7f6bd90f4700 1 -- 172.21.15.27:0/1436713182 learned_addr learned my addr 172.21.15.27:0/1436713182 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:25.527 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.523+0000 7f6bd3fff700 1 --2- 172.21.15.27:0/1436713182 >> v2:172.21.15.124:3300/0 conn(0x7f6bd40d1030 0x7f6bd40c66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:25.527 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.523+0000 7f6bd88f3700 1 --2- 172.21.15.27:0/1436713182 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6bd40cdc00 0x7f6bd40cbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:25.527 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.524+0000 7f6bd90f4700 1 -- 172.21.15.27:0/1436713182 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6bd40cdc00 msgr2=0x7f6bd40cbe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:25.527 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.524+0000 7f6bd90f4700 1 --2- 172.21.15.27:0/1436713182 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6bd40cdc00 0x7f6bd40cbe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:25.527 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.524+0000 7f6bd90f4700 1 -- 172.21.15.27:0/1436713182 >> v2:172.21.15.124:3300/0 conn(0x7f6bd40d1030 msgr2=0x7f6bd40c66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:25.527 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.524+0000 7f6bd90f4700 1 --2- 172.21.15.27:0/1436713182 >> v2:172.21.15.124:3300/0 conn(0x7f6bd40d1030 0x7f6bd40c66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:25.527 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.524+0000 7f6bd90f4700 1 -- 172.21.15.27:0/1436713182 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd40ad530 con 0x7f6bd40c6bf0 2023-10-06T01:42:25.527 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.524+0000 7f6bd90f4700 1 --2- 172.21.15.27:0/1436713182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd40c6bf0 0x7f6bd40ae5d0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4089820 tx=0x7f6bcc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db13df90ff29441f server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:25.527 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.524+0000 7f6bd37fe700 1 -- 172.21.15.27:0/1436713182 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc00b640 con 0x7f6bd40c6bf0 2023-10-06T01:42:25.527 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.524+0000 7f6bd37fe700 1 -- 172.21.15.27:0/1436713182 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bcc00b7a0 con 0x7f6bd40c6bf0 2023-10-06T01:42:25.528 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.524+0000 7f6bd37fe700 1 -- 172.21.15.27:0/1436713182 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bcc00bb40 con 0x7f6bd40c6bf0 2023-10-06T01:42:25.528 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.524+0000 7f6bdab57700 1 -- 172.21.15.27:0/1436713182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd40c6bf0 msgr2=0x7f6bd40ae5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:25.528 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.524+0000 7f6bdab57700 1 --2- 172.21.15.27:0/1436713182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd40c6bf0 0x7f6bd40ae5d0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4089820 tx=0x7f6bcc009250 comp rx=0 tx=0).stop 2023-10-06T01:42:25.528 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.525+0000 7f6bdab57700 1 -- 172.21.15.27:0/1436713182 shutdown_connections 2023-10-06T01:42:25.528 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.525+0000 7f6bdab57700 1 --2- 172.21.15.27:0/1436713182 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd40c6bf0 0x7f6bd40ae5d0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:25.528 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.525+0000 7f6bdab57700 1 --2- 172.21.15.27:0/1436713182 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6bd40cdc00 0x7f6bd40cbe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:25.528 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.525+0000 7f6bdab57700 1 --2- 172.21.15.27:0/1436713182 >> v2:172.21.15.124:3300/0 conn(0x7f6bd40d1030 0x7f6bd40c66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:25.528 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.525+0000 7f6bdab57700 1 -- 172.21.15.27:0/1436713182 >> 172.21.15.27:0/1436713182 conn(0x7f6bd4099990 msgr2=0x7f6bd40b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:25.528 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.525+0000 7f6bdab57700 1 -- 172.21.15.27:0/1436713182 shutdown_connections 2023-10-06T01:42:25.528 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.525+0000 7f6bdab57700 1 -- 172.21.15.27:0/1436713182 wait complete. 2023-10-06T01:42:25.531 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.526+0000 7f6bdab57700 1 Processor -- start 2023-10-06T01:42:25.531 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.526+0000 7f6bdab57700 1 -- start start 2023-10-06T01:42:25.531 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.526+0000 7f6bdab57700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6bd40c6bf0 0x7f6bd4067630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:25.531 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.526+0000 7f6bdab57700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd40cdc00 0x7f6bd40c9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:25.531 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.527+0000 7f6bdab57700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6bd40d1030 0x7f6bd4168660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:25.531 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.527+0000 7f6bdab57700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd40686b0 con 0x7f6bd40cdc00 2023-10-06T01:42:25.531 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.527+0000 7f6bdab57700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd409a4c0 con 0x7f6bd40d1030 2023-10-06T01:42:25.531 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.527+0000 7f6bdab57700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd4067b70 con 0x7f6bd40c6bf0 2023-10-06T01:42:25.531 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.527+0000 7f6bd3fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd40cdc00 0x7f6bd40c9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:25.531 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.527+0000 7f6bd3fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd40cdc00 0x7f6bd40c9ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48228/0 (socket says 172.21.15.27:48228) 2023-10-06T01:42:25.532 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.527+0000 7f6bd3fff700 1 -- 172.21.15.27:0/2498369441 learned_addr learned my addr 172.21.15.27:0/2498369441 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:25.532 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.527+0000 7f6bd88f3700 1 --2- 172.21.15.27:0/2498369441 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6bd40c6bf0 0x7f6bd4067630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:25.532 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.527+0000 7f6bd90f4700 1 --2- 172.21.15.27:0/2498369441 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6bd40d1030 0x7f6bd4168660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:25.532 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.527+0000 7f6bd3fff700 1 -- 172.21.15.27:0/2498369441 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6bd40c6bf0 msgr2=0x7f6bd4067630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:25.532 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.527+0000 7f6bd3fff700 1 --2- 172.21.15.27:0/2498369441 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6bd40c6bf0 0x7f6bd4067630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:25.532 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.527+0000 7f6bd3fff700 1 -- 172.21.15.27:0/2498369441 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6bd40d1030 msgr2=0x7f6bd4168660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:25.532 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.527+0000 7f6bd3fff700 1 --2- 172.21.15.27:0/2498369441 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6bd40d1030 0x7f6bd4168660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:25.532 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.527+0000 7f6bd3fff700 1 -- 172.21.15.27:0/2498369441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bcc0094f0 con 0x7f6bd40cdc00 2023-10-06T01:42:25.532 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.527+0000 7f6bd3fff700 1 --2- 172.21.15.27:0/2498369441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd40cdc00 0x7f6bd40c9ab0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f6bc4009e50 tx=0x7f6bc400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:25.532 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.527+0000 7f6bd0ff9700 1 -- 172.21.15.27:0/2498369441 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc400caa0 con 0x7f6bd40cdc00 2023-10-06T01:42:25.532 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.527+0000 7f6bd0ff9700 1 -- 172.21.15.27:0/2498369441 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bc400f040 con 0x7f6bd40cdc00 2023-10-06T01:42:25.532 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.527+0000 7f6bdab57700 1 -- 172.21.15.27:0/2498369441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bd4067e10 con 0x7f6bd40cdc00 2023-10-06T01:42:25.533 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.527+0000 7f6bd0ff9700 1 -- 172.21.15.27:0/2498369441 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bc400ce30 con 0x7f6bd40cdc00 2023-10-06T01:42:25.533 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.527+0000 7f6bdab57700 1 -- 172.21.15.27:0/2498369441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bd40683d0 con 0x7f6bd40cdc00 2023-10-06T01:42:25.533 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.528+0000 7f6bdab57700 1 -- 172.21.15.27:0/2498369441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bd4064050 con 0x7f6bd40cdc00 2023-10-06T01:42:25.533 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.529+0000 7f6bd0ff9700 1 -- 172.21.15.27:0/2498369441 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6bc4014690 con 0x7f6bd40cdc00 2023-10-06T01:42:25.533 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.529+0000 7f6bd0ff9700 1 --2- 172.21.15.27:0/2498369441 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6bb4066f30 0x7f6bb40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:25.533 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.529+0000 7f6bd0ff9700 1 -- 172.21.15.27:0/2498369441 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6453+0+0 (secure 0 0 0) 0x7f6bc40853c0 con 0x7f6bd40cdc00 2023-10-06T01:42:25.533 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.529+0000 7f6bd88f3700 1 --2- 172.21.15.27:0/2498369441 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6bb4066f30 0x7f6bb40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:25.533 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.530+0000 7f6bd88f3700 1 --2- 172.21.15.27:0/2498369441 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6bb4066f30 0x7f6bb40693e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc009da0 tx=0x7f6bbc009120 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:25.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.533+0000 7f6bd0ff9700 1 -- 172.21.15.27:0/2498369441 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6bc4052340 con 0x7f6bd40cdc00 2023-10-06T01:42:25.728 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.726+0000 7f6bdab57700 1 -- 172.21.15.27:0/2498369441 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6bd4063ae0 con 0x7f6bd40cdc00 2023-10-06T01:42:25.728 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.727+0000 7f6bd0ff9700 1 -- 172.21.15.27:0/2498369441 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+18425 (secure 0 0 0) 0x7f6bc4055960 con 0x7f6bd40cdc00 2023-10-06T01:42:25.729 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:25.729 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":34,"fsid":"20a7bff0-74f8-4f2f-b6c6-899e804bc965","created":"2023-10-06T01:40:54.474215+0000","modified":"2023-10-06T01:42:18.365024+0000","last_up_change":"2023-10-06T01:41:05.242847+0000","last_in_change":"2023-10-06T01:40:56.890960+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-10-06T01:41:05.685386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-10-06T01:41:06.092995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2023-10-06T01:42:01.295053+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2023-10-06T01:42:02.005152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}},{"pool":7,"pool_name":"data_new","create_time":"2023-10-06T01:42:16.105156+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"b902c86d-800c-40c5-b152-b545e50d02c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":90644},{"type":"v1","addr":"172.21.15.27:6801","nonce":90644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":90644},{"type":"v1","addr":"172.21.15.27:6803","nonce":90644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":90644},{"type":"v1","addr":"172.21.15.27:6807","nonce":90644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":90644},{"type":"v1","addr":"172.21.15.27:6805","nonce":90644}]},"public_addr":"172.21.15.27:6801/90644","cluster_addr":"172.21.15.27:6803/90644","heartbeat_back_addr":"172.21.15.27:6807/90644","heartbeat_front_addr":"172.21.15.27:6805/90644","state":["exists","up"]},{"osd":1,"uuid":"5b4a70e5-b06e-48cd-bfc4-7962c5d05386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":90643},{"type":"v1","addr":"172.21.15.27:6809","nonce":90643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":90643},{"type":"v1","addr":"172.21.15.27:6811","nonce":90643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":90643},{"type":"v1","addr":"172.21.15.27:6815","nonce":90643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":90643},{"type":"v1","addr":"172.21.15.27:6813","nonce":90643}]},"public_addr":"172.21.15.27:6809/90643","cluster_addr":"172.21.15.27:6811/90643","heartbeat_back_addr":"172.21.15.27:6815/90643","heartbeat_front_addr":"172.21.15.27:6813/90643","state":["exists","up"]},{"osd":2,"uuid":"22d9de46-4073-4336-9122-5227e601eca0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":90645},{"type":"v1","addr":"172.21.15.27:6817","nonce":90645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":90645},{"type":"v1","addr":"172.21.15.27:6819","nonce":90645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":90645},{"type":"v1","addr":"172.21.15.27:6823","nonce":90645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":90645},{"type":"v1","addr":"172.21.15.27:6821","nonce":90645}]},"public_addr":"172.21.15.27:6817/90645","cluster_addr":"172.21.15.27:6819/90645","heartbeat_back_addr":"172.21.15.27:6823/90645","heartbeat_front_addr":"172.21.15.27:6821/90645","state":["exists","up"]},{"osd":3,"uuid":"438bb2d3-34e2-4b7c-9734-8360c1b67650","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":90668},{"type":"v1","addr":"172.21.15.27:6825","nonce":90668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":90668},{"type":"v1","addr":"172.21.15.27:6827","nonce":90668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":90668},{"type":"v1","addr":"172.21.15.27:6831","nonce":90668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":90668},{"type":"v1","addr":"172.21.15.27:6829","nonce":90668}]},"public_addr":"172.21.15.27:6825/90668","cluster_addr":"172.21.15.27:6827/90668","heartbeat_back_addr":"172.21.15.27:6831/90668","heartbeat_front_addr":"172.21.15.27:6829/90668","state":["exists","up"]},{"osd":4,"uuid":"e1956a8e-e0f8-4e7c-acbb-40037415a354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":90863},{"type":"v1","addr":"172.21.15.124:6801","nonce":90863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":90863},{"type":"v1","addr":"172.21.15.124:6803","nonce":90863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":90863},{"type":"v1","addr":"172.21.15.124:6807","nonce":90863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":90863},{"type":"v1","addr":"172.21.15.124:6805","nonce":90863}]},"public_addr":"172.21.15.124:6801/90863","cluster_addr":"172.21.15.124:6803/90863","heartbeat_back_addr":"172.21.15.124:6807/90863","heartbeat_front_addr":"172.21.15.124:6805/90863","state":["exists","up"]},{"osd":5,"uuid":"306c59d4-846d-4b73-b1c9-051bec9e9368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":90885},{"type":"v1","addr":"172.21.15.124:6817","nonce":90885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":90885},{"type":"v1","addr":"172.21.15.124:6819","nonce":90885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":90885},{"type":"v1","addr":"172.21.15.124:6823","nonce":90885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":90885},{"type":"v1","addr":"172.21.15.124:6821","nonce":90885}]},"public_addr":"172.21.15.124:6817/90885","cluster_addr":"172.21.15.124:6819/90885","heartbeat_back_addr":"172.21.15.124:6823/90885","heartbeat_front_addr":"172.21.15.124:6821/90885","state":["exists","up"]},{"osd":6,"uuid":"15463e53-ac8d-47d1-b4f6-c0d31f29e4a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":90864},{"type":"v1","addr":"172.21.15.124:6809","nonce":90864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":90864},{"type":"v1","addr":"172.21.15.124:6811","nonce":90864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":90864},{"type":"v1","addr":"172.21.15.124:6815","nonce":90864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":90864},{"type":"v1","addr":"172.21.15.124:6813","nonce":90864}]},"public_addr":"172.21.15.124:6809/90864","cluster_addr":"172.21.15.124:6811/90864","heartbeat_back_addr":"172.21.15.124:6815/90864","heartbeat_front_addr":"172.21.15.124:6813/90864","state":["exists","up"]},{"osd":7,"uuid":"ae463a90-a9d3-498f-86fa-47a35b3fe889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":90894},{"type":"v1","addr":"172.21.15.124:6825","nonce":90894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":90894},{"type":"v1","addr":"172.21.15.124:6827","nonce":90894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":90894},{"type":"v1","addr":"172.21.15.124:6831","nonce":90894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":90894},{"type":"v1","addr":"172.21.15.124:6829","nonce":90894}]},"public_addr":"172.21.15.124:6825/90894","cluster_addr":"172.21.15.124:6827/90894","heartbeat_back_addr":"172.21.15.124:6831/90894","heartbeat_front_addr":"172.21.15.124:6829/90894","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.827704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.839951+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.032230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.050605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.987908+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.031290+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.080707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.053692+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T01:42:25.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.729+0000 7f6bdab57700 1 -- 172.21.15.27:0/2498369441 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6bb4066f30 msgr2=0x7f6bb40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:25.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.729+0000 7f6bdab57700 1 --2- 172.21.15.27:0/2498369441 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6bb4066f30 0x7f6bb40693e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc009da0 tx=0x7f6bbc009120 comp rx=0 tx=0).stop 2023-10-06T01:42:25.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.729+0000 7f6bdab57700 1 -- 172.21.15.27:0/2498369441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd40cdc00 msgr2=0x7f6bd40c9ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:25.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.729+0000 7f6bdab57700 1 --2- 172.21.15.27:0/2498369441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd40cdc00 0x7f6bd40c9ab0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f6bc4009e50 tx=0x7f6bc400b040 comp rx=0 tx=0).stop 2023-10-06T01:42:25.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.729+0000 7f6bdab57700 1 -- 172.21.15.27:0/2498369441 shutdown_connections 2023-10-06T01:42:25.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.729+0000 7f6bdab57700 1 --2- 172.21.15.27:0/2498369441 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6bb4066f30 0x7f6bb40693e0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:25.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.729+0000 7f6bdab57700 1 --2- 172.21.15.27:0/2498369441 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6bd40c6bf0 0x7f6bd4067630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:25.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.729+0000 7f6bdab57700 1 --2- 172.21.15.27:0/2498369441 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6bd40d1030 0x7f6bd4168660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:25.731 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.729+0000 7f6bdab57700 1 --2- 172.21.15.27:0/2498369441 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6bd40cdc00 0x7f6bd40c9ab0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:25.732 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.729+0000 7f6bdab57700 1 -- 172.21.15.27:0/2498369441 >> 172.21.15.27:0/2498369441 conn(0x7f6bd4099990 msgr2=0x7f6bd40d0870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:25.732 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.729+0000 7f6bdab57700 1 -- 172.21.15.27:0/2498369441 shutdown_connections 2023-10-06T01:42:25.732 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.729+0000 7f6bdab57700 1 -- 172.21.15.27:0/2498369441 wait complete. 2023-10-06T01:42:25.743 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-06T01:42:25.862 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.861+0000 7fbffec70700 1 Processor -- start 2023-10-06T01:42:25.862 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.861+0000 7fbffec70700 1 -- start start 2023-10-06T01:42:25.865 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.862+0000 7fbffec70700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbff8142ec0 0x7fbff80cf040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:25.865 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.862+0000 7fbffec70700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbff8143290 0x7fbff80cf580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:25.865 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.862+0000 7fbffec70700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fbff80682b0 0x7fbff8159d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:25.865 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.862+0000 7fbffec70700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbff8067e20 con 0x7fbff8142ec0 2023-10-06T01:42:25.865 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.862+0000 7fbffec70700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbff815ab10 con 0x7fbff80682b0 2023-10-06T01:42:25.865 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.862+0000 7fbffec70700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbff80677d0 con 0x7fbff8143290 2023-10-06T01:42:25.865 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.862+0000 7fbffdc6e700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbff8142ec0 0x7fbff80cf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:25.865 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.862+0000 7fbffdc6e700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbff8142ec0 0x7fbff80cf040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48230/0 (socket says 172.21.15.27:48230) 2023-10-06T01:42:25.865 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.862+0000 7fbffdc6e700 1 -- 172.21.15.27:0/1792013983 learned_addr learned my addr 172.21.15.27:0/1792013983 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:25.865 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.862+0000 7fbffe46f700 1 --2- 172.21.15.27:0/1792013983 >> v2:172.21.15.124:3300/0 conn(0x7fbff80682b0 0x7fbff8159d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:25.865 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.862+0000 7fbffd46d700 1 --2- 172.21.15.27:0/1792013983 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbff8143290 0x7fbff80cf580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:25.866 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.862+0000 7fbffdc6e700 1 -- 172.21.15.27:0/1792013983 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbff8143290 msgr2=0x7fbff80cf580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:25.866 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.862+0000 7fbffdc6e700 1 --2- 172.21.15.27:0/1792013983 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbff8143290 0x7fbff80cf580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:25.866 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.862+0000 7fbffdc6e700 1 -- 172.21.15.27:0/1792013983 >> v2:172.21.15.124:3300/0 conn(0x7fbff80682b0 msgr2=0x7fbff8159d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:25.866 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.862+0000 7fbffdc6e700 1 --2- 172.21.15.27:0/1792013983 >> v2:172.21.15.124:3300/0 conn(0x7fbff80682b0 0x7fbff8159d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:25.866 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.862+0000 7fbffdc6e700 1 -- 172.21.15.27:0/1792013983 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff8067ab0 con 0x7fbff8142ec0 2023-10-06T01:42:25.866 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.863+0000 7fbffdc6e700 1 --2- 172.21.15.27:0/1792013983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbff8142ec0 0x7fbff80cf040 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fbff806a1d0 tx=0x7fbff0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2331d2c8ab6e7ee server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:25.866 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.863+0000 7fbffcc6c700 1 -- 172.21.15.27:0/1792013983 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff0009e70 con 0x7fbff8142ec0 2023-10-06T01:42:25.866 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.863+0000 7fbffcc6c700 1 -- 172.21.15.27:0/1792013983 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbff000b860 con 0x7fbff8142ec0 2023-10-06T01:42:25.866 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.863+0000 7fbffcc6c700 1 -- 172.21.15.27:0/1792013983 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff000bc00 con 0x7fbff8142ec0 2023-10-06T01:42:25.866 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.863+0000 7fbffec70700 1 -- 172.21.15.27:0/1792013983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbff8142ec0 msgr2=0x7fbff80cf040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:25.866 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.863+0000 7fbffec70700 1 --2- 172.21.15.27:0/1792013983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbff8142ec0 0x7fbff80cf040 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fbff806a1d0 tx=0x7fbff0009250 comp rx=0 tx=0).stop 2023-10-06T01:42:25.867 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.863+0000 7fbffec70700 1 -- 172.21.15.27:0/1792013983 shutdown_connections 2023-10-06T01:42:25.867 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.863+0000 7fbffec70700 1 --2- 172.21.15.27:0/1792013983 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbff8142ec0 0x7fbff80cf040 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:25.867 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.863+0000 7fbffec70700 1 --2- 172.21.15.27:0/1792013983 >> v2:172.21.15.124:3300/0 conn(0x7fbff80682b0 0x7fbff8159d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:25.867 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.863+0000 7fbffec70700 1 --2- 172.21.15.27:0/1792013983 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbff8143290 0x7fbff80cf580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:25.867 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.863+0000 7fbffec70700 1 -- 172.21.15.27:0/1792013983 >> 172.21.15.27:0/1792013983 conn(0x7fbff80ccf70 msgr2=0x7fbff80cd370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:25.867 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.863+0000 7fbffec70700 1 -- 172.21.15.27:0/1792013983 shutdown_connections 2023-10-06T01:42:25.867 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.863+0000 7fbffec70700 1 -- 172.21.15.27:0/1792013983 wait complete. 2023-10-06T01:42:25.869 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.864+0000 7fbffec70700 1 Processor -- start 2023-10-06T01:42:25.869 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.864+0000 7fbffec70700 1 -- start start 2023-10-06T01:42:25.869 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.865+0000 7fbffec70700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbff80682b0 0x7fbff80b3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:25.869 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.865+0000 7fbffec70700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbff8142ec0 0x7fbff80c7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.865+0000 7fbffec70700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbff8143290 0x7fbff8130e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.865+0000 7fbffec70700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff80b40f0 con 0x7fbff8142ec0 2023-10-06T01:42:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.865+0000 7fbffec70700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff80b43d0 con 0x7fbff80682b0 2023-10-06T01:42:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.865+0000 7fbffec70700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbff80b46b0 con 0x7fbff8143290 2023-10-06T01:42:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.865+0000 7fbffd46d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbff8142ec0 0x7fbff80c7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.865+0000 7fbffd46d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbff8142ec0 0x7fbff80c7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48236/0 (socket says 172.21.15.27:48236) 2023-10-06T01:42:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.865+0000 7fbffd46d700 1 -- 172.21.15.27:0/3631526393 learned_addr learned my addr 172.21.15.27:0/3631526393 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.865+0000 7fbffdc6e700 1 --2- 172.21.15.27:0/3631526393 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbff80682b0 0x7fbff80b3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.866+0000 7fbffd46d700 1 -- 172.21.15.27:0/3631526393 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbff8143290 msgr2=0x7fbff8130e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.866+0000 7fbffd46d700 1 --2- 172.21.15.27:0/3631526393 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbff8143290 0x7fbff8130e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:25.870 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.866+0000 7fbffd46d700 1 -- 172.21.15.27:0/3631526393 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbff80682b0 msgr2=0x7fbff80b3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:25.871 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.866+0000 7fbffd46d700 1 --2- 172.21.15.27:0/3631526393 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbff80682b0 0x7fbff80b3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:25.871 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.866+0000 7fbffd46d700 1 -- 172.21.15.27:0/3631526393 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff00094f0 con 0x7fbff8142ec0 2023-10-06T01:42:25.871 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.866+0000 7fbffdc6e700 1 --2- 172.21.15.27:0/3631526393 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbff80682b0 0x7fbff80b3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:42:25.871 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.866+0000 7fbffd46d700 1 --2- 172.21.15.27:0/3631526393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbff8142ec0 0x7fbff80c7c20 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fbfec00d860 tx=0x7fbfec008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:25.871 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.866+0000 7fbff5ffb700 1 -- 172.21.15.27:0/3631526393 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfec009610 con 0x7fbff8142ec0 2023-10-06T01:42:25.871 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.866+0000 7fbff5ffb700 1 -- 172.21.15.27:0/3631526393 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfec009770 con 0x7fbff8142ec0 2023-10-06T01:42:25.871 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.866+0000 7fbffec70700 1 -- 172.21.15.27:0/3631526393 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbff80b4990 con 0x7fbff8142ec0 2023-10-06T01:42:25.871 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.866+0000 7fbff5ffb700 1 -- 172.21.15.27:0/3631526393 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfec009b30 con 0x7fbff8142ec0 2023-10-06T01:42:25.871 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.866+0000 7fbffec70700 1 -- 172.21.15.27:0/3631526393 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbff812e630 con 0x7fbff8142ec0 2023-10-06T01:42:25.871 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.867+0000 7fbffec70700 1 -- 172.21.15.27:0/3631526393 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbff80b4e30 con 0x7fbff8142ec0 2023-10-06T01:42:25.871 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.867+0000 7fbff5ffb700 1 -- 172.21.15.27:0/3631526393 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fbfec01e460 con 0x7fbff8142ec0 2023-10-06T01:42:25.871 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.868+0000 7fbff5ffb700 1 --2- 172.21.15.27:0/3631526393 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fbfe4066f30 0x7fbfe40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.868+0000 7fbff5ffb700 1 -- 172.21.15.27:0/3631526393 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6453+0+0 (secure 0 0 0) 0x7fbfec084f30 con 0x7fbff8142ec0 2023-10-06T01:42:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.868+0000 7fbffdc6e700 1 --2- 172.21.15.27:0/3631526393 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fbfe4066f30 0x7fbfe40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:25.872 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.869+0000 7fbffdc6e700 1 --2- 172.21.15.27:0/3631526393 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fbfe4066f30 0x7fbfe40693e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbff806a1d0 tx=0x7fbff0017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:25.873 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:25.872+0000 7fbff5ffb700 1 -- 172.21.15.27:0/3631526393 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbfec051e10 con 0x7fbff8142ec0 2023-10-06T01:42:26.110 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.108+0000 7fbffec70700 1 -- 172.21.15.27:0/3631526393 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fbff804c3c0 con 0x7fbff8142ec0 2023-10-06T01:42:26.110 INFO:tasks.ceph.mon.a.smithi027.stderr:2023-10-06T01:42:26.109+0000 7f4bb5c52700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-06T01:42:26.116 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.115+0000 7fbff5ffb700 1 -- 172.21.15.27:0/3631526393 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v17) v1 ==== 161+0+0 (secure 0 0 0) 0x7fbfec055430 con 0x7fbff8142ec0 2023-10-06T01:42:26.119 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.117+0000 7fbffec70700 1 -- 172.21.15.27:0/3631526393 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fbfe4066f30 msgr2=0x7fbfe40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:26.119 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.117+0000 7fbffec70700 1 --2- 172.21.15.27:0/3631526393 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fbfe4066f30 0x7fbfe40693e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbff806a1d0 tx=0x7fbff0017000 comp rx=0 tx=0).stop 2023-10-06T01:42:26.119 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.117+0000 7fbffec70700 1 -- 172.21.15.27:0/3631526393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbff8142ec0 msgr2=0x7fbff80c7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:26.119 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.117+0000 7fbffec70700 1 --2- 172.21.15.27:0/3631526393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbff8142ec0 0x7fbff80c7c20 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fbfec00d860 tx=0x7fbfec008040 comp rx=0 tx=0).stop 2023-10-06T01:42:26.119 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.117+0000 7fbffec70700 1 -- 172.21.15.27:0/3631526393 shutdown_connections 2023-10-06T01:42:26.119 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.117+0000 7fbffec70700 1 --2- 172.21.15.27:0/3631526393 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fbfe4066f30 0x7fbfe40693e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:26.120 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.117+0000 7fbffec70700 1 --2- 172.21.15.27:0/3631526393 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbff8143290 0x7fbff8130e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:26.120 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.117+0000 7fbffec70700 1 --2- 172.21.15.27:0/3631526393 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbff80682b0 0x7fbff80b3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:26.120 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.117+0000 7fbffec70700 1 --2- 172.21.15.27:0/3631526393 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbff8142ec0 0x7fbff80c7c20 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:26.120 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.117+0000 7fbffec70700 1 -- 172.21.15.27:0/3631526393 >> 172.21.15.27:0/3631526393 conn(0x7fbff80ccf70 msgr2=0x7fbff8142090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:26.120 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.117+0000 7fbffec70700 1 -- 172.21.15.27:0/3631526393 shutdown_connections 2023-10-06T01:42:26.120 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.117+0000 7fbffec70700 1 -- 172.21.15.27:0/3631526393 wait complete. 2023-10-06T01:42:26.120 INFO:teuthology.orchestra.run.smithi027.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-06T01:42:26.121 INFO:tasks.ceph.mds.d.smithi124.stderr: -8437> 2023-10-06T01:42:07.616+0000 7f17014a6700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-06T01:42:26.130 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-06T01:42:26.150 INFO:tasks.ceph.mds.d.smithi124.stdout:starting mds.d at 2023-10-06T01:42:26.251 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.250+0000 7f62e5ad2700 1 Processor -- start 2023-10-06T01:42:26.252 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.250+0000 7f62e5ad2700 1 -- start start 2023-10-06T01:42:26.253 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.251+0000 7f62e5ad2700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f62e00cdd00 0x7f62e0157f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:26.253 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.251+0000 7f62e5ad2700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62e0159a60 con 0x7f62e0157b50 2023-10-06T01:42:26.254 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.251+0000 7f62e5ad2700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62e0158e50 con 0x7f62e0158460 2023-10-06T01:42:26.254 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.251+0000 7f62e5ad2700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62e0159130 con 0x7f62e00cdd00 2023-10-06T01:42:26.254 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.251+0000 7f62df7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f62e0157b50 0x7f62e00cd5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51226/0 (socket says 172.21.15.27:51226) 2023-10-06T01:42:26.254 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.251+0000 7f62df7fe700 1 -- 172.21.15.27:0/3679308202 learned_addr learned my addr 172.21.15.27:0/3679308202 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:26.254 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.251+0000 7f62deffd700 1 --2- 172.21.15.27:0/3679308202 >> v2:172.21.15.27:3300/0 conn(0x7f62e00cdd00 0x7f62e0157f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:26.254 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.252+0000 7f62deffd700 1 -- 172.21.15.27:0/3679308202 >> v1:172.21.15.124:6789/0 conn(0x7f62e0158460 legacy=0x7f62e0161110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:26.255 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.252+0000 7f62deffd700 1 -- 172.21.15.27:0/3679308202 >> v1:172.21.15.27:6789/0 conn(0x7f62e0157b50 legacy=0x7f62e00cd5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:26.255 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.252+0000 7f62deffd700 1 -- 172.21.15.27:0/3679308202 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e0159410 con 0x7f62e00cdd00 2023-10-06T01:42:26.255 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.252+0000 7f62de7fc700 1 -- 172.21.15.27:0/3679308202 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2228369361 0 0) 0x7f62e0159a60 con 0x7f62e0157b50 2023-10-06T01:42:26.255 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.252+0000 7f62deffd700 1 --2- 172.21.15.27:0/3679308202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62e00cdd00 0x7f62e0157f20 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f62c80098e0 tx=0x7f62c80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6300b543faf7df74 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:26.255 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.252+0000 7f62de7fc700 1 -- 172.21.15.27:0/3679308202 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c8009e80 con 0x7f62e00cdd00 2023-10-06T01:42:26.255 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.252+0000 7f62de7fc700 1 -- 172.21.15.27:0/3679308202 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62c800b860 con 0x7f62e00cdd00 2023-10-06T01:42:26.256 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.252+0000 7f62de7fc700 1 -- 172.21.15.27:0/3679308202 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c800bbe0 con 0x7f62e00cdd00 2023-10-06T01:42:26.259 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.252+0000 7f62e5ad2700 1 -- 172.21.15.27:0/3679308202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62e00cdd00 msgr2=0x7f62e0157f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:26.259 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.252+0000 7f62e5ad2700 1 --2- 172.21.15.27:0/3679308202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62e00cdd00 0x7f62e0157f20 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f62c80098e0 tx=0x7f62c80091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:26.259 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.253+0000 7f62e5ad2700 1 -- 172.21.15.27:0/3679308202 shutdown_connections 2023-10-06T01:42:26.259 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.253+0000 7f62e5ad2700 1 --2- 172.21.15.27:0/3679308202 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62e00cdd00 0x7f62e0157f20 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:26.259 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.253+0000 7f62e5ad2700 1 -- 172.21.15.27:0/3679308202 >> 172.21.15.27:0/3679308202 conn(0x7f62e006d890 msgr2=0x7f62e006dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:26.260 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.253+0000 7f62e5ad2700 1 -- 172.21.15.27:0/3679308202 shutdown_connections 2023-10-06T01:42:26.260 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.253+0000 7f62e5ad2700 1 -- 172.21.15.27:0/3679308202 wait complete. 2023-10-06T01:42:26.260 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.254+0000 7f62e5ad2700 1 Processor -- start 2023-10-06T01:42:26.260 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.254+0000 7f62e5ad2700 1 -- start start 2023-10-06T01:42:26.260 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.255+0000 7f62e5ad2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62e00cdd00 0x7f62e00c9b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:26.260 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.255+0000 7f62e5ad2700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f62e0157b50 0x7f62e0152170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:26.260 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.255+0000 7f62e5ad2700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f62e0158460 0x7f62e015b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:26.260 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.255+0000 7f62e5ad2700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e0153650 con 0x7f62e00cdd00 2023-10-06T01:42:26.260 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.255+0000 7f62e5ad2700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e0153930 con 0x7f62e0157b50 2023-10-06T01:42:26.261 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.255+0000 7f62e5ad2700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62e00ca0a0 con 0x7f62e0158460 2023-10-06T01:42:26.261 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.255+0000 7f62df7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62e00cdd00 0x7f62e00c9b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:26.261 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.255+0000 7f62df7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62e00cdd00 0x7f62e00c9b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48250/0 (socket says 172.21.15.27:48250) 2023-10-06T01:42:26.261 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.255+0000 7f62deffd700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f62e0157b50 0x7f62e0152170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:26.261 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.255+0000 7f62df7fe700 1 -- 172.21.15.27:0/2871944227 learned_addr learned my addr 172.21.15.27:0/2871944227 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:26.261 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.255+0000 7f62dffff700 1 --2- 172.21.15.27:0/2871944227 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f62e0158460 0x7f62e015b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:26.261 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.255+0000 7f62df7fe700 1 -- 172.21.15.27:0/2871944227 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f62e0158460 msgr2=0x7f62e015b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:26.261 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.255+0000 7f62df7fe700 1 --2- 172.21.15.27:0/2871944227 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f62e0158460 0x7f62e015b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:26.261 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.255+0000 7f62df7fe700 1 -- 172.21.15.27:0/2871944227 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f62e0157b50 msgr2=0x7f62e0152170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:26.261 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.255+0000 7f62df7fe700 1 --2- 172.21.15.27:0/2871944227 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f62e0157b50 0x7f62e0152170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:26.261 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.255+0000 7f62df7fe700 1 -- 172.21.15.27:0/2871944227 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c8009420 con 0x7f62e00cdd00 2023-10-06T01:42:26.261 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.255+0000 7f62deffd700 1 --2- 172.21.15.27:0/2871944227 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f62e0157b50 0x7f62e0152170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:42:26.262 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.255+0000 7f62df7fe700 1 --2- 172.21.15.27:0/2871944227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62e00cdd00 0x7f62e00c9b60 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f62d000c870 tx=0x7f62d0007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:26.262 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.255+0000 7f62d77fe700 1 -- 172.21.15.27:0/2871944227 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d0005c00 con 0x7f62e00cdd00 2023-10-06T01:42:26.262 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.255+0000 7f62d77fe700 1 -- 172.21.15.27:0/2871944227 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62d0005d60 con 0x7f62e00cdd00 2023-10-06T01:42:26.262 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.255+0000 7f62e5ad2700 1 -- 172.21.15.27:0/2871944227 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62e0152ae0 con 0x7f62e00cdd00 2023-10-06T01:42:26.262 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.255+0000 7f62d77fe700 1 -- 172.21.15.27:0/2871944227 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62d0007320 con 0x7f62e00cdd00 2023-10-06T01:42:26.262 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.256+0000 7f62e5ad2700 1 -- 172.21.15.27:0/2871944227 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62e01530a0 con 0x7f62e00cdd00 2023-10-06T01:42:26.262 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.257+0000 7f62e5ad2700 1 -- 172.21.15.27:0/2871944227 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62e0064050 con 0x7f62e00cdd00 2023-10-06T01:42:26.262 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.257+0000 7f62d77fe700 1 -- 172.21.15.27:0/2871944227 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f62d001b3b0 con 0x7f62e00cdd00 2023-10-06T01:42:26.262 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.257+0000 7f62d77fe700 1 --2- 172.21.15.27:0/2871944227 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f62c0066f30 0x7f62c00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:26.262 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.257+0000 7f62d77fe700 1 -- 172.21.15.27:0/2871944227 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 6539+0+0 (secure 0 0 0) 0x7f62d0083c80 con 0x7f62e00cdd00 2023-10-06T01:42:26.262 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.257+0000 7f62deffd700 1 --2- 172.21.15.27:0/2871944227 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f62c0066f30 0x7f62c00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:26.263 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.258+0000 7f62deffd700 1 --2- 172.21.15.27:0/2871944227 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f62c0066f30 0x7f62c00693e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f62c80093f0 tx=0x7f62c8017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:26.263 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.261+0000 7f62d77fe700 1 -- 172.21.15.27:0/2871944227 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f62d0050bb0 con 0x7f62e00cdd00 2023-10-06T01:42:26.500 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:26.499+0000 7f62e5ad2700 1 -- 172.21.15.27:0/2871944227 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f62e0158240 con 0x7f62e00cdd00 2023-10-06T01:42:27.115 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.114+0000 7f62d77fe700 1 -- 172.21.15.27:0/2871944227 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v18) v1 ==== 104+0+0 (secure 0 0 0) 0x7f62d00541d0 con 0x7f62e00cdd00 2023-10-06T01:42:27.118 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.116+0000 7f62e5ad2700 1 -- 172.21.15.27:0/2871944227 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f62c0066f30 msgr2=0x7f62c00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:27.118 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.116+0000 7f62e5ad2700 1 --2- 172.21.15.27:0/2871944227 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f62c0066f30 0x7f62c00693e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f62c80093f0 tx=0x7f62c8017000 comp rx=0 tx=0).stop 2023-10-06T01:42:27.118 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.116+0000 7f62e5ad2700 1 -- 172.21.15.27:0/2871944227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62e00cdd00 msgr2=0x7f62e00c9b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:27.118 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.116+0000 7f62e5ad2700 1 --2- 172.21.15.27:0/2871944227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62e00cdd00 0x7f62e00c9b60 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f62d000c870 tx=0x7f62d0007d90 comp rx=0 tx=0).stop 2023-10-06T01:42:27.118 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.116+0000 7f62e5ad2700 1 -- 172.21.15.27:0/2871944227 shutdown_connections 2023-10-06T01:42:27.118 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.116+0000 7f62e5ad2700 1 --2- 172.21.15.27:0/2871944227 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f62c0066f30 0x7f62c00693e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:27.118 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.116+0000 7f62e5ad2700 1 --2- 172.21.15.27:0/2871944227 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f62e0158460 0x7f62e015b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:27.118 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.116+0000 7f62e5ad2700 1 --2- 172.21.15.27:0/2871944227 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f62e00cdd00 0x7f62e00c9b60 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:27.118 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.116+0000 7f62e5ad2700 1 --2- 172.21.15.27:0/2871944227 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f62e0157b50 0x7f62e0152170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:27.119 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.116+0000 7f62e5ad2700 1 -- 172.21.15.27:0/2871944227 >> 172.21.15.27:0/2871944227 conn(0x7f62e006d890 msgr2=0x7f62e014ef60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:27.119 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.116+0000 7f62e5ad2700 1 -- 172.21.15.27:0/2871944227 shutdown_connections 2023-10-06T01:42:27.119 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.116+0000 7f62e5ad2700 1 -- 172.21.15.27:0/2871944227 wait complete. 2023-10-06T01:42:27.131 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-06T01:42:27.257 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.256+0000 7f305d05b700 1 Processor -- start 2023-10-06T01:42:27.257 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.256+0000 7f305d05b700 1 -- start start 2023-10-06T01:42:27.259 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.257+0000 7f305d05b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f30580ccbe0 0x7f30580b5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:27.259 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.257+0000 7f305d05b700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30580cbed0 con 0x7f30580d0f00 2023-10-06T01:42:27.260 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.257+0000 7f305d05b700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30580cc750 con 0x7f30580b55b0 2023-10-06T01:42:27.260 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.257+0000 7f305d05b700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3058159d40 con 0x7f30580ccbe0 2023-10-06T01:42:27.260 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.257+0000 7f305659c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f30580ccbe0 0x7f30580b5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:27.260 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.257+0000 7f305759e700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f30580d0f00 0x7f30580ce270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51242/0 (socket says 172.21.15.27:51242) 2023-10-06T01:42:27.260 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.257+0000 7f305759e700 1 -- 172.21.15.27:0/2711050664 learned_addr learned my addr 172.21.15.27:0/2711050664 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:27.260 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.257+0000 7f305659c700 1 -- 172.21.15.27:0/2711050664 >> v1:172.21.15.124:6789/0 conn(0x7f30580b55b0 legacy=0x7f30580cf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:27.260 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.257+0000 7f305659c700 1 -- 172.21.15.27:0/2711050664 >> v1:172.21.15.27:6789/0 conn(0x7f30580d0f00 legacy=0x7f30580ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:27.260 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.257+0000 7f305659c700 1 -- 172.21.15.27:0/2711050664 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30580cd050 con 0x7f30580ccbe0 2023-10-06T01:42:27.260 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.257+0000 7f3055d9b700 1 -- 172.21.15.27:0/2711050664 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1061402498 0 0) 0x7f30580cbed0 con 0x7f30580d0f00 2023-10-06T01:42:27.260 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.257+0000 7f305659c700 1 --2- 172.21.15.27:0/2711050664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30580ccbe0 0x7f30580b5980 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f304c0098e0 tx=0x7f304c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8e37cd07aa42452 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:27.261 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.257+0000 7f3055d9b700 1 -- 172.21.15.27:0/2711050664 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304c009e80 con 0x7f30580ccbe0 2023-10-06T01:42:27.261 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.257+0000 7f3055d9b700 1 -- 172.21.15.27:0/2711050664 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f304c00b860 con 0x7f30580ccbe0 2023-10-06T01:42:27.261 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.257+0000 7f3055d9b700 1 -- 172.21.15.27:0/2711050664 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304c00bbe0 con 0x7f30580ccbe0 2023-10-06T01:42:27.261 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.258+0000 7f305d05b700 1 -- 172.21.15.27:0/2711050664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30580ccbe0 msgr2=0x7f30580b5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:27.261 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.258+0000 7f305d05b700 1 --2- 172.21.15.27:0/2711050664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30580ccbe0 0x7f30580b5980 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f304c0098e0 tx=0x7f304c0091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:27.261 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.258+0000 7f305d05b700 1 -- 172.21.15.27:0/2711050664 shutdown_connections 2023-10-06T01:42:27.261 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.258+0000 7f305d05b700 1 --2- 172.21.15.27:0/2711050664 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30580ccbe0 0x7f30580b5980 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:27.261 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.258+0000 7f305d05b700 1 -- 172.21.15.27:0/2711050664 >> 172.21.15.27:0/2711050664 conn(0x7f305808a050 msgr2=0x7f3058099970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:27.261 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.258+0000 7f305d05b700 1 -- 172.21.15.27:0/2711050664 shutdown_connections 2023-10-06T01:42:27.261 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.258+0000 7f305d05b700 1 -- 172.21.15.27:0/2711050664 wait complete. 2023-10-06T01:42:27.264 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.259+0000 7f305d05b700 1 Processor -- start 2023-10-06T01:42:27.264 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.259+0000 7f305d05b700 1 -- start start 2023-10-06T01:42:27.264 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.260+0000 7f305d05b700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f30580b55b0 0x7f30581699a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:27.264 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.260+0000 7f305d05b700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f30580ccbe0 0x7f305816c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:27.264 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.260+0000 7f305d05b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30580d0f00 0x7f305815c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:27.264 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.260+0000 7f305d05b700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30580b1a00 con 0x7f30580d0f00 2023-10-06T01:42:27.264 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.260+0000 7f305d05b700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30580b0e60 con 0x7f30580ccbe0 2023-10-06T01:42:27.264 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.260+0000 7f305d05b700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30580b10d0 con 0x7f30580b55b0 2023-10-06T01:42:27.265 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.260+0000 7f305759e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30580d0f00 0x7f305815c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:27.265 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.260+0000 7f305759e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30580d0f00 0x7f305815c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48272/0 (socket says 172.21.15.27:48272) 2023-10-06T01:42:27.265 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.260+0000 7f305759e700 1 -- 172.21.15.27:0/4182370709 learned_addr learned my addr 172.21.15.27:0/4182370709 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:27.265 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.260+0000 7f305659c700 1 --2- 172.21.15.27:0/4182370709 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f30580ccbe0 0x7f305816c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:27.265 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.260+0000 7f3056d9d700 1 --2- 172.21.15.27:0/4182370709 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f30580b55b0 0x7f30581699a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:27.265 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.261+0000 7f305759e700 1 -- 172.21.15.27:0/4182370709 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f30580b55b0 msgr2=0x7f30581699a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:27.265 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.261+0000 7f305759e700 1 --2- 172.21.15.27:0/4182370709 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f30580b55b0 0x7f30581699a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:27.265 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.261+0000 7f305759e700 1 -- 172.21.15.27:0/4182370709 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f30580ccbe0 msgr2=0x7f305816c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:27.265 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.261+0000 7f305759e700 1 --2- 172.21.15.27:0/4182370709 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f30580ccbe0 0x7f305816c650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:27.265 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.261+0000 7f305759e700 1 -- 172.21.15.27:0/4182370709 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f304c009420 con 0x7f30580d0f00 2023-10-06T01:42:27.265 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.261+0000 7f3056d9d700 1 --2- 172.21.15.27:0/4182370709 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f30580b55b0 0x7f30581699a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:42:27.265 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.261+0000 7f305759e700 1 --2- 172.21.15.27:0/4182370709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30580d0f00 0x7f305815c8c0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f305000cc20 tx=0x7f30500019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:27.266 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.261+0000 7f303effd700 1 -- 172.21.15.27:0/4182370709 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3050004bb0 con 0x7f30580d0f00 2023-10-06T01:42:27.266 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.261+0000 7f303effd700 1 -- 172.21.15.27:0/4182370709 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3050004d10 con 0x7f30580d0f00 2023-10-06T01:42:27.266 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.261+0000 7f305d05b700 1 -- 172.21.15.27:0/4182370709 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30580b13b0 con 0x7f30580d0f00 2023-10-06T01:42:27.266 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.261+0000 7f303effd700 1 -- 172.21.15.27:0/4182370709 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305000ab20 con 0x7f30580d0f00 2023-10-06T01:42:27.266 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.261+0000 7f305d05b700 1 -- 172.21.15.27:0/4182370709 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f305816cb90 con 0x7f30580d0f00 2023-10-06T01:42:27.266 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.262+0000 7f305d05b700 1 -- 172.21.15.27:0/4182370709 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30580717c0 con 0x7f30580d0f00 2023-10-06T01:42:27.266 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.262+0000 7f303effd700 1 -- 172.21.15.27:0/4182370709 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f30500070f0 con 0x7f30580d0f00 2023-10-06T01:42:27.266 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.263+0000 7f303effd700 1 --2- 172.21.15.27:0/4182370709 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3044066f30 0x7f30440693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:27.266 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.263+0000 7f303effd700 1 -- 172.21.15.27:0/4182370709 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 6539+0+0 (secure 0 0 0) 0x7f30500846a0 con 0x7f30580d0f00 2023-10-06T01:42:27.266 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.263+0000 7f3056d9d700 1 --2- 172.21.15.27:0/4182370709 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3044066f30 0x7f30440693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:27.267 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.263+0000 7f3056d9d700 1 --2- 172.21.15.27:0/4182370709 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3044066f30 0x7f30440693e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f30480018f0 tx=0x7f3048009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:27.268 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.267+0000 7f303effd700 1 -- 172.21.15.27:0/4182370709 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f30500515b0 con 0x7f30580d0f00 2023-10-06T01:42:27.464 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:27.462+0000 7f305d05b700 1 -- 172.21.15.27:0/4182370709 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f30580b4a30 con 0x7f30580d0f00 2023-10-06T01:42:28.419 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.416+0000 7f303effd700 1 -- 172.21.15.27:0/4182370709 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v36) v1 ==== 181+0+0 (secure 0 0 0) 0x7f30500180d0 con 0x7f30580d0f00 2023-10-06T01:42:28.421 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.419+0000 7f305d05b700 1 -- 172.21.15.27:0/4182370709 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3044066f30 msgr2=0x7f30440693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:28.421 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.419+0000 7f305d05b700 1 --2- 172.21.15.27:0/4182370709 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3044066f30 0x7f30440693e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f30480018f0 tx=0x7f3048009040 comp rx=0 tx=0).stop 2023-10-06T01:42:28.421 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.419+0000 7f305d05b700 1 -- 172.21.15.27:0/4182370709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30580d0f00 msgr2=0x7f305815c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:28.421 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.419+0000 7f305d05b700 1 --2- 172.21.15.27:0/4182370709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30580d0f00 0x7f305815c8c0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f305000cc20 tx=0x7f30500019b0 comp rx=0 tx=0).stop 2023-10-06T01:42:28.421 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.419+0000 7f305d05b700 1 -- 172.21.15.27:0/4182370709 shutdown_connections 2023-10-06T01:42:28.421 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.419+0000 7f305d05b700 1 --2- 172.21.15.27:0/4182370709 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3044066f30 0x7f30440693e0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:28.421 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.419+0000 7f305d05b700 1 --2- 172.21.15.27:0/4182370709 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f30580b55b0 0x7f30581699a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:28.422 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.419+0000 7f305d05b700 1 --2- 172.21.15.27:0/4182370709 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f30580d0f00 0x7f305815c8c0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:28.422 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.419+0000 7f305d05b700 1 --2- 172.21.15.27:0/4182370709 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f30580ccbe0 0x7f305816c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:28.423 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.419+0000 7f305d05b700 1 -- 172.21.15.27:0/4182370709 >> 172.21.15.27:0/4182370709 conn(0x7f305808a050 msgr2=0x7f30580ad170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:28.423 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.420+0000 7f305d05b700 1 -- 172.21.15.27:0/4182370709 shutdown_connections 2023-10-06T01:42:28.423 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.420+0000 7f305d05b700 1 -- 172.21.15.27:0/4182370709 wait complete. 2023-10-06T01:42:28.424 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_metadata' removed 2023-10-06T01:42:28.438 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-06T01:42:28.568 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.565+0000 7f4478de8700 1 Processor -- start 2023-10-06T01:42:28.568 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.565+0000 7f4478de8700 1 -- start start 2023-10-06T01:42:28.570 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.566+0000 7f4478de8700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f44740cc000 0x7f44740cc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:28.570 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.566+0000 7f4478de8700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4474069ec0 con 0x7f4474142d00 2023-10-06T01:42:28.570 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.566+0000 7f4478de8700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44740cc930 con 0x7f44741430d0 2023-10-06T01:42:28.570 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.566+0000 7f4478de8700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4474068fd0 con 0x7f44740cc000 2023-10-06T01:42:28.571 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.566+0000 7f44737fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f4474142d00 0x7f447406a350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51252/0 (socket says 172.21.15.27:51252) 2023-10-06T01:42:28.571 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.566+0000 7f4473fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f44740cc000 0x7f44740cc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:28.571 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.566+0000 7f44737fe700 1 -- 172.21.15.27:0/1841993775 learned_addr learned my addr 172.21.15.27:0/1841993775 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:28.571 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.566+0000 7f4473fff700 1 -- 172.21.15.27:0/1841993775 >> v1:172.21.15.124:6789/0 conn(0x7f44741430d0 legacy=0x7f44740c41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:28.571 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.566+0000 7f4473fff700 1 -- 172.21.15.27:0/1841993775 >> v1:172.21.15.27:6789/0 conn(0x7f4474142d00 legacy=0x7f447406a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:28.571 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.566+0000 7f4473fff700 1 -- 172.21.15.27:0/1841993775 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44740692b0 con 0x7f44740cc000 2023-10-06T01:42:28.571 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.566+0000 7f44727fc700 1 -- 172.21.15.27:0/1841993775 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 732816720 0 0) 0x7f4474069ec0 con 0x7f4474142d00 2023-10-06T01:42:28.571 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.567+0000 7f4473fff700 1 --2- 172.21.15.27:0/1841993775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44740cc000 0x7f44740cc3f0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f446c009e60 tx=0x7f446c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=184222f75b031c94 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:28.572 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.567+0000 7f44727fc700 1 -- 172.21.15.27:0/1841993775 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f446c00e070 con 0x7f44740cc000 2023-10-06T01:42:28.572 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.567+0000 7f44727fc700 1 -- 172.21.15.27:0/1841993775 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f446c00bc30 con 0x7f44740cc000 2023-10-06T01:42:28.572 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.567+0000 7f44727fc700 1 -- 172.21.15.27:0/1841993775 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f446c012430 con 0x7f44740cc000 2023-10-06T01:42:28.572 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.567+0000 7f4478de8700 1 -- 172.21.15.27:0/1841993775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44740cc000 msgr2=0x7f44740cc3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:28.572 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.567+0000 7f4478de8700 1 --2- 172.21.15.27:0/1841993775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44740cc000 0x7f44740cc3f0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f446c009e60 tx=0x7f446c009780 comp rx=0 tx=0).stop 2023-10-06T01:42:28.572 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.567+0000 7f4478de8700 1 -- 172.21.15.27:0/1841993775 shutdown_connections 2023-10-06T01:42:28.572 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.567+0000 7f4478de8700 1 --2- 172.21.15.27:0/1841993775 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f44740cc000 0x7f44740cc3f0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:28.572 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.567+0000 7f4478de8700 1 -- 172.21.15.27:0/1841993775 >> 172.21.15.27:0/1841993775 conn(0x7f44740ce430 msgr2=0x7f44740ce830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:28.572 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.567+0000 7f4478de8700 1 -- 172.21.15.27:0/1841993775 shutdown_connections 2023-10-06T01:42:28.572 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.567+0000 7f4478de8700 1 -- 172.21.15.27:0/1841993775 wait complete. 2023-10-06T01:42:28.575 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.569+0000 7f4478de8700 1 Processor -- start 2023-10-06T01:42:28.576 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.569+0000 7f4478de8700 1 -- start start 2023-10-06T01:42:28.576 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.569+0000 7f4478de8700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f44740cc000 0x7f4474134cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:28.576 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.569+0000 7f4478de8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4474142d00 0x7f447415e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:28.576 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.569+0000 7f4478de8700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f44741430d0 0x7f447415e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:28.576 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.569+0000 7f4478de8700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44741309c0 con 0x7f4474142d00 2023-10-06T01:42:28.576 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.569+0000 7f4478de8700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f447412fdb0 con 0x7f44740cc000 2023-10-06T01:42:28.576 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.569+0000 7f4478de8700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4474130090 con 0x7f44741430d0 2023-10-06T01:42:28.576 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.569+0000 7f4472ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4474142d00 0x7f447415e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:28.576 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.569+0000 7f4472ffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4474142d00 0x7f447415e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48288/0 (socket says 172.21.15.27:48288) 2023-10-06T01:42:28.576 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.569+0000 7f4472ffd700 1 -- 172.21.15.27:0/658179286 learned_addr learned my addr 172.21.15.27:0/658179286 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:28.577 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.570+0000 7f4473fff700 1 --2- 172.21.15.27:0/658179286 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f44741430d0 0x7f447415e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:28.577 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.570+0000 7f44737fe700 1 --2- 172.21.15.27:0/658179286 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f44740cc000 0x7f4474134cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:28.577 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.570+0000 7f4472ffd700 1 -- 172.21.15.27:0/658179286 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f44741430d0 msgr2=0x7f447415e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:28.577 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.570+0000 7f4472ffd700 1 --2- 172.21.15.27:0/658179286 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f44741430d0 0x7f447415e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:28.577 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.570+0000 7f4472ffd700 1 -- 172.21.15.27:0/658179286 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f44740cc000 msgr2=0x7f4474134cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:28.577 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.570+0000 7f4472ffd700 1 --2- 172.21.15.27:0/658179286 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f44740cc000 0x7f4474134cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:28.577 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.570+0000 7f4472ffd700 1 -- 172.21.15.27:0/658179286 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f446c009a20 con 0x7f4474142d00 2023-10-06T01:42:28.577 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.570+0000 7f4472ffd700 1 --2- 172.21.15.27:0/658179286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4474142d00 0x7f447415e030 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f446800b5e0 tx=0x7f4468009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:28.577 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.570+0000 7f445b7fe700 1 -- 172.21.15.27:0/658179286 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f446800f510 con 0x7f4474142d00 2023-10-06T01:42:28.578 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.570+0000 7f445b7fe700 1 -- 172.21.15.27:0/658179286 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f446800f670 con 0x7f4474142d00 2023-10-06T01:42:28.578 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.570+0000 7f4478de8700 1 -- 172.21.15.27:0/658179286 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4474130370 con 0x7f4474142d00 2023-10-06T01:42:28.578 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.570+0000 7f445b7fe700 1 -- 172.21.15.27:0/658179286 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f446800fa30 con 0x7f4474142d00 2023-10-06T01:42:28.578 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.570+0000 7f4478de8700 1 -- 172.21.15.27:0/658179286 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44741323f0 con 0x7f4474142d00 2023-10-06T01:42:28.578 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.571+0000 7f4478de8700 1 -- 172.21.15.27:0/658179286 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4474130b30 con 0x7f4474142d00 2023-10-06T01:42:28.578 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.571+0000 7f445b7fe700 1 -- 172.21.15.27:0/658179286 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f446801a400 con 0x7f4474142d00 2023-10-06T01:42:28.578 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.572+0000 7f445b7fe700 1 --2- 172.21.15.27:0/658179286 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4460066f30 0x7f44600693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:28.578 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.572+0000 7f445b7fe700 1 -- 172.21.15.27:0/658179286 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 6101+0+0 (secure 0 0 0) 0x7f44680841f0 con 0x7f4474142d00 2023-10-06T01:42:28.578 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.572+0000 7f44737fe700 1 --2- 172.21.15.27:0/658179286 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4460066f30 0x7f44600693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:28.578 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.572+0000 7f44737fe700 1 --2- 172.21.15.27:0/658179286 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4460066f30 0x7f44600693e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f44640011e0 tx=0x7f4464009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:28.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.577+0000 7f445b7fe700 1 -- 172.21.15.27:0/658179286 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4468051350 con 0x7f4474142d00 2023-10-06T01:42:28.771 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:28.770+0000 7f4478de8700 1 -- 172.21.15.27:0/658179286 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4474064050 con 0x7f4474142d00 2023-10-06T01:42:29.420 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.420+0000 7f445b7fe700 1 -- 172.21.15.27:0/658179286 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v37) v1 ==== 169+0+0 (secure 0 0 0) 0x7f4468054970 con 0x7f4474142d00 2023-10-06T01:42:29.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.429+0000 7f4478de8700 1 -- 172.21.15.27:0/658179286 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4460066f30 msgr2=0x7f44600693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:29.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.429+0000 7f4478de8700 1 --2- 172.21.15.27:0/658179286 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4460066f30 0x7f44600693e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f44640011e0 tx=0x7f4464009040 comp rx=0 tx=0).stop 2023-10-06T01:42:29.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.429+0000 7f4478de8700 1 -- 172.21.15.27:0/658179286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4474142d00 msgr2=0x7f447415e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:29.429 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.429+0000 7f4478de8700 1 --2- 172.21.15.27:0/658179286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4474142d00 0x7f447415e030 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f446800b5e0 tx=0x7f4468009120 comp rx=0 tx=0).stop 2023-10-06T01:42:29.430 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.429+0000 7f4478de8700 1 -- 172.21.15.27:0/658179286 shutdown_connections 2023-10-06T01:42:29.430 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.429+0000 7f4478de8700 1 --2- 172.21.15.27:0/658179286 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f4460066f30 0x7f44600693e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:29.430 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.429+0000 7f4478de8700 1 --2- 172.21.15.27:0/658179286 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f44741430d0 0x7f447415e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:29.430 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.429+0000 7f4478de8700 1 --2- 172.21.15.27:0/658179286 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f44740cc000 0x7f4474134cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:29.430 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.429+0000 7f4478de8700 1 --2- 172.21.15.27:0/658179286 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f4474142d00 0x7f447415e030 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:29.430 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.429+0000 7f4478de8700 1 -- 172.21.15.27:0/658179286 >> 172.21.15.27:0/658179286 conn(0x7f44740ce430 msgr2=0x7f4474141bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:29.431 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.431+0000 7f4478de8700 1 -- 172.21.15.27:0/658179286 shutdown_connections 2023-10-06T01:42:29.431 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.431+0000 7f4478de8700 1 -- 172.21.15.27:0/658179286 wait complete. 2023-10-06T01:42:29.432 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'cephfs_data' removed 2023-10-06T01:42:29.445 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm data_new data_new --yes-i-really-really-mean-it 2023-10-06T01:42:29.575 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.575+0000 7f3fc3989700 1 Processor -- start 2023-10-06T01:42:29.575 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.575+0000 7f3fc3989700 1 -- start start 2023-10-06T01:42:29.578 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.576+0000 7f3fc3989700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3fbc15b840 0x7f3fbc06f350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:29.578 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.576+0000 7f3fc3989700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f3fbc15bc10 0x7f3fbc0b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:29.578 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.576+0000 7f3fc3989700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fbc0c7a10 con 0x7f3fbc15c080 2023-10-06T01:42:29.578 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.576+0000 7f3fc3989700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fbc0c8590 con 0x7f3fbc15bc10 2023-10-06T01:42:29.578 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.576+0000 7f3fc3989700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc0c7c60 con 0x7f3fbc15b840 2023-10-06T01:42:29.578 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.576+0000 7f3fc1725700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3fbc15b840 0x7f3fbc06f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:29.578 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.576+0000 7f3fc0f24700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f3fbc15bc10 0x7f3fbc0b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:29.578 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.576+0000 7f3fc1725700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3fbc15b840 0x7f3fbc06f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.27:45754/0 (socket says 172.21.15.27:45754) 2023-10-06T01:42:29.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.576+0000 7f3fc1f26700 1 --1- >> v1:172.21.15.124:6789/0 conn(0x7f3fbc15c080 0x7f3fbc0ae580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.27:60354/0 (socket says 172.21.15.27:60354) 2023-10-06T01:42:29.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.576+0000 7f3fc0f24700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f3fbc15bc10 0x7f3fbc0b0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.27:41454/0 (socket says 172.21.15.27:41454) 2023-10-06T01:42:29.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.576+0000 7f3fc1725700 1 -- 172.21.15.27:0/2649577835 learned_addr learned my addr 172.21.15.27:0/2649577835 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:29.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.576+0000 7f3fc0f24700 1 -- 172.21.15.27:0/2649577835 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3fbc15b840 msgr2=0x7f3fbc06f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:29.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.576+0000 7f3fc0f24700 1 --2- 172.21.15.27:0/2649577835 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3fbc15b840 0x7f3fbc06f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:29.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.576+0000 7f3fc0f24700 1 -- 172.21.15.27:0/2649577835 >> v1:172.21.15.124:6789/0 conn(0x7f3fbc15c080 legacy=0x7f3fbc0ae580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:29.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.576+0000 7f3fc0f24700 1 -- 172.21.15.27:0/2649577835 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fbc0c7f40 con 0x7f3fbc15bc10 2023-10-06T01:42:29.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.576+0000 7f3fb3fff700 1 -- 172.21.15.27:0/2649577835 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 887532898 0 0) 0x7f3fbc0c7a10 con 0x7f3fbc15c080 2023-10-06T01:42:29.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.576+0000 7f3fc1725700 1 --2- 172.21.15.27:0/2649577835 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3fbc15b840 0x7f3fbc06f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:42:29.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.577+0000 7f3fc0f24700 1 --2- 172.21.15.27:0/2649577835 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3fbc15bc10 0x7f3fbc0b0bb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3fac0098e0 tx=0x7f3fac0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a6343d30c68495d server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:29.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.577+0000 7f3fb3fff700 1 -- 172.21.15.27:0/2649577835 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fac00b640 con 0x7f3fbc15bc10 2023-10-06T01:42:29.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.577+0000 7f3fb3fff700 1 -- 172.21.15.27:0/2649577835 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fac00b7a0 con 0x7f3fbc15bc10 2023-10-06T01:42:29.580 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.577+0000 7f3fb3fff700 1 -- 172.21.15.27:0/2649577835 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fac00bb40 con 0x7f3fbc15bc10 2023-10-06T01:42:29.580 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.577+0000 7f3fc3989700 1 -- 172.21.15.27:0/2649577835 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3fbc15bc10 msgr2=0x7f3fbc0b0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:29.580 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.577+0000 7f3fc3989700 1 --2- 172.21.15.27:0/2649577835 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3fbc15bc10 0x7f3fbc0b0bb0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3fac0098e0 tx=0x7f3fac0091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:29.580 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.577+0000 7f3fc3989700 1 -- 172.21.15.27:0/2649577835 shutdown_connections 2023-10-06T01:42:29.580 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.577+0000 7f3fc3989700 1 --2- 172.21.15.27:0/2649577835 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3fbc15b840 0x7f3fbc06f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:29.580 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.577+0000 7f3fc3989700 1 --2- 172.21.15.27:0/2649577835 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3fbc15bc10 0x7f3fbc0b0bb0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:29.580 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.577+0000 7f3fc3989700 1 -- 172.21.15.27:0/2649577835 >> 172.21.15.27:0/2649577835 conn(0x7f3fbc06e7d0 msgr2=0x7f3fbc06ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:29.580 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.577+0000 7f3fc3989700 1 -- 172.21.15.27:0/2649577835 shutdown_connections 2023-10-06T01:42:29.580 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.578+0000 7f3fc3989700 1 -- 172.21.15.27:0/2649577835 wait complete. 2023-10-06T01:42:29.583 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.579+0000 7f3fc3989700 1 Processor -- start 2023-10-06T01:42:29.583 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.579+0000 7f3fc3989700 1 -- start start 2023-10-06T01:42:29.583 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.579+0000 7f3fc3989700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc15b840 0x7f3fbc0c3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:29.583 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.579+0000 7f3fc3989700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3fbc15bc10 0x7f3fbc0cbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:29.583 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.579+0000 7f3fc3989700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3fbc0c2190 0x7f3fbc0b0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:29.583 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.579+0000 7f3fc3989700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc152ba0 con 0x7f3fbc15b840 2023-10-06T01:42:29.583 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.579+0000 7f3fc3989700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc0c2690 con 0x7f3fbc0c2190 2023-10-06T01:42:29.583 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.579+0000 7f3fc3989700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc0c2970 con 0x7f3fbc15bc10 2023-10-06T01:42:29.583 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.580+0000 7f3fc1725700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc15b840 0x7f3fbc0c3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:29.583 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.580+0000 7f3fc0f24700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3fbc15bc10 0x7f3fbc0cbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:29.583 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.580+0000 7f3fc1725700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc15b840 0x7f3fbc0c3730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48294/0 (socket says 172.21.15.27:48294) 2023-10-06T01:42:29.584 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.580+0000 7f3fc1f26700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3fbc0c2190 0x7f3fbc0b0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:29.584 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.580+0000 7f3fc1725700 1 -- 172.21.15.27:0/1398336145 learned_addr learned my addr 172.21.15.27:0/1398336145 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:29.584 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.580+0000 7f3fc1725700 1 -- 172.21.15.27:0/1398336145 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3fbc15bc10 msgr2=0x7f3fbc0cbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:29.584 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.580+0000 7f3fc1725700 1 --2- 172.21.15.27:0/1398336145 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3fbc15bc10 0x7f3fbc0cbbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:29.584 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.580+0000 7f3fc1725700 1 -- 172.21.15.27:0/1398336145 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3fbc0c2190 msgr2=0x7f3fbc0b0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:29.584 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.580+0000 7f3fc1725700 1 --2- 172.21.15.27:0/1398336145 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3fbc0c2190 0x7f3fbc0b0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:29.584 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.580+0000 7f3fc1725700 1 -- 172.21.15.27:0/1398336145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb80094f0 con 0x7f3fbc15b840 2023-10-06T01:42:29.584 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.580+0000 7f3fc1f26700 1 --2- 172.21.15.27:0/1398336145 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3fbc0c2190 0x7f3fbc0b0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:42:29.584 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.580+0000 7f3fc1725700 1 --2- 172.21.15.27:0/1398336145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc15b840 0x7f3fbc0c3730 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f3fb800ec50 tx=0x7f3fb8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:29.584 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.580+0000 7f3fb17fa700 1 -- 172.21.15.27:0/1398336145 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb800c850 con 0x7f3fbc15b840 2023-10-06T01:42:29.584 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.580+0000 7f3fb17fa700 1 -- 172.21.15.27:0/1398336145 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fb800c9b0 con 0x7f3fbc15b840 2023-10-06T01:42:29.585 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.580+0000 7f3fc3989700 1 -- 172.21.15.27:0/1398336145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fac009420 con 0x7f3fbc15b840 2023-10-06T01:42:29.585 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.580+0000 7f3fb17fa700 1 -- 172.21.15.27:0/1398336145 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fb800cd30 con 0x7f3fbc15b840 2023-10-06T01:42:29.585 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.580+0000 7f3fc3989700 1 -- 172.21.15.27:0/1398336145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fbc0c2f30 con 0x7f3fbc15b840 2023-10-06T01:42:29.585 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.581+0000 7f3fc3989700 1 -- 172.21.15.27:0/1398336145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fbc0c3580 con 0x7f3fbc15b840 2023-10-06T01:42:29.585 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.582+0000 7f3fb17fa700 1 -- 172.21.15.27:0/1398336145 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f3fb8013680 con 0x7f3fbc15b840 2023-10-06T01:42:29.585 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.582+0000 7f3fb17fa700 1 --2- 172.21.15.27:0/1398336145 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3fa0066f30 0x7f3fa00693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:29.585 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.582+0000 7f3fb17fa700 1 -- 172.21.15.27:0/1398336145 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5687+0+0 (secure 0 0 0) 0x7f3fb8084770 con 0x7f3fbc15b840 2023-10-06T01:42:29.585 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.582+0000 7f3fc0f24700 1 --2- 172.21.15.27:0/1398336145 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3fa0066f30 0x7f3fa00693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:29.585 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.583+0000 7f3fc0f24700 1 --2- 172.21.15.27:0/1398336145 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3fa0066f30 0x7f3fa00693e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3fac0093f0 tx=0x7f3fac017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:29.586 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.586+0000 7f3fb17fa700 1 -- 172.21.15.27:0/1398336145 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3fb80519f0 con 0x7f3fbc15b840 2023-10-06T01:42:29.779 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:29.779+0000 7f3fc3989700 1 -- 172.21.15.27:0/1398336145 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "data_new", "pool2": "data_new", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3fbc0717c0 con 0x7f3fbc15b840 2023-10-06T01:42:30.435 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.434+0000 7f3fb17fa700 1 -- 172.21.15.27:0/1398336145 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "data_new", "pool2": "data_new", "yes_i_really_really_mean_it": true}]=0 pool 'data_new' removed v38) v1 ==== 160+0+0 (secure 0 0 0) 0x7f3fb8055010 con 0x7f3fbc15b840 2023-10-06T01:42:30.438 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.437+0000 7f3fc3989700 1 -- 172.21.15.27:0/1398336145 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3fa0066f30 msgr2=0x7f3fa00693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:30.438 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.437+0000 7f3fc3989700 1 --2- 172.21.15.27:0/1398336145 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3fa0066f30 0x7f3fa00693e0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f3fac0093f0 tx=0x7f3fac017000 comp rx=0 tx=0).stop 2023-10-06T01:42:30.438 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.437+0000 7f3fc3989700 1 -- 172.21.15.27:0/1398336145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc15b840 msgr2=0x7f3fbc0c3730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:30.438 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.437+0000 7f3fc3989700 1 --2- 172.21.15.27:0/1398336145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc15b840 0x7f3fbc0c3730 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f3fb800ec50 tx=0x7f3fb8009250 comp rx=0 tx=0).stop 2023-10-06T01:42:30.439 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.438+0000 7f3fc3989700 1 -- 172.21.15.27:0/1398336145 shutdown_connections 2023-10-06T01:42:30.439 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.438+0000 7f3fc3989700 1 --2- 172.21.15.27:0/1398336145 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f3fa0066f30 0x7f3fa00693e0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:30.439 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.438+0000 7f3fc3989700 1 --2- 172.21.15.27:0/1398336145 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f3fbc0c2190 0x7f3fbc0b0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:30.439 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.438+0000 7f3fc3989700 1 --2- 172.21.15.27:0/1398336145 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f3fbc15b840 0x7f3fbc0c3730 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:30.439 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.438+0000 7f3fc3989700 1 --2- 172.21.15.27:0/1398336145 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f3fbc15bc10 0x7f3fbc0cbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:30.439 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.438+0000 7f3fc3989700 1 -- 172.21.15.27:0/1398336145 >> 172.21.15.27:0/1398336145 conn(0x7f3fbc06e7d0 msgr2=0x7f3fbc159580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:30.440 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.438+0000 7f3fc3989700 1 -- 172.21.15.27:0/1398336145 shutdown_connections 2023-10-06T01:42:30.440 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.438+0000 7f3fc3989700 1 -- 172.21.15.27:0/1398336145 wait complete. 2023-10-06T01:42:30.440 INFO:teuthology.orchestra.run.smithi027.stderr:pool 'data_new' removed 2023-10-06T01:42:30.452 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mds mds_symlink_recovery 2023-10-06T01:42:30.576 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.575+0000 7fae7c01d700 1 Processor -- start 2023-10-06T01:42:30.576 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.576+0000 7fae7c01d700 1 -- start start 2023-10-06T01:42:30.578 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.576+0000 7fae7c01d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fae74159500 0x7fae741598d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:30.578 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.576+0000 7fae7c01d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fae7415dab0 0x7fae740cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:30.578 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.576+0000 7fae7c01d700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae740b1110 con 0x7fae740b3fd0 2023-10-06T01:42:30.578 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.576+0000 7fae7c01d700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae740b13b0 con 0x7fae7415dab0 2023-10-06T01:42:30.578 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.576+0000 7fae7c01d700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae740b1690 con 0x7fae74159500 2023-10-06T01:42:30.578 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.576+0000 7fae7a5ba700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fae7415dab0 0x7fae740cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:30.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.577+0000 7fae7a5ba700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fae7415dab0 0x7fae740cf6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48296/0 (socket says 172.21.15.27:48296) 2023-10-06T01:42:30.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.577+0000 7fae7a5ba700 1 -- 172.21.15.27:0/2704184085 learned_addr learned my addr 172.21.15.27:0/2704184085 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:30.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.577+0000 7fae79db9700 1 --2- 172.21.15.27:0/2704184085 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fae74159500 0x7fae741598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:30.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.577+0000 7fae7a5ba700 1 -- 172.21.15.27:0/2704184085 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fae74159500 msgr2=0x7fae741598d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:30.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.577+0000 7fae7a5ba700 1 --2- 172.21.15.27:0/2704184085 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fae74159500 0x7fae741598d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:30.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.577+0000 7fae7a5ba700 1 -- 172.21.15.27:0/2704184085 >> v1:172.21.15.124:6789/0 conn(0x7fae740b3fd0 legacy=0x7fae7415d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:30.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.577+0000 7fae7a5ba700 1 -- 172.21.15.27:0/2704184085 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae740b1970 con 0x7fae7415dab0 2023-10-06T01:42:30.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.577+0000 7fae7a5ba700 1 --2- 172.21.15.27:0/2704184085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae7415dab0 0x7fae740cf6c0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fae6c0099b0 tx=0x7fae6c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13429d60a7da1c server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:30.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.577+0000 7fae78db7700 1 -- 172.21.15.27:0/2704184085 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6c009ea0 con 0x7fae7415dab0 2023-10-06T01:42:30.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.577+0000 7fae78db7700 1 -- 172.21.15.27:0/2704184085 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae6c00b860 con 0x7fae7415dab0 2023-10-06T01:42:30.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.577+0000 7fae78db7700 1 -- 172.21.15.27:0/2704184085 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6c00bc00 con 0x7fae7415dab0 2023-10-06T01:42:30.579 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.578+0000 7fae7c01d700 1 -- 172.21.15.27:0/2704184085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae7415dab0 msgr2=0x7fae740cf6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:30.580 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.578+0000 7fae7c01d700 1 --2- 172.21.15.27:0/2704184085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae7415dab0 0x7fae740cf6c0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fae6c0099b0 tx=0x7fae6c009250 comp rx=0 tx=0).stop 2023-10-06T01:42:30.580 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.578+0000 7fae7c01d700 1 -- 172.21.15.27:0/2704184085 shutdown_connections 2023-10-06T01:42:30.580 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.578+0000 7fae7c01d700 1 --2- 172.21.15.27:0/2704184085 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae7415dab0 0x7fae740cf6c0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:30.580 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.578+0000 7fae7c01d700 1 --2- 172.21.15.27:0/2704184085 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fae74159500 0x7fae741598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:30.580 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.578+0000 7fae7c01d700 1 -- 172.21.15.27:0/2704184085 >> 172.21.15.27:0/2704184085 conn(0x7fae74067630 msgr2=0x7fae74069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:30.580 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.578+0000 7fae7c01d700 1 -- 172.21.15.27:0/2704184085 shutdown_connections 2023-10-06T01:42:30.580 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.578+0000 7fae7c01d700 1 -- 172.21.15.27:0/2704184085 wait complete. 2023-10-06T01:42:30.582 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.579+0000 7fae7c01d700 1 Processor -- start 2023-10-06T01:42:30.582 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.579+0000 7fae7c01d700 1 -- start start 2023-10-06T01:42:30.582 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.580+0000 7fae7c01d700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fae740b3fd0 0x7fae74153fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:30.582 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.580+0000 7fae7c01d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae74159500 0x7fae74149210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:30.583 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.580+0000 7fae7c01d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fae7415dab0 0x7fae74143d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:30.583 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.580+0000 7fae7c01d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae74145410 con 0x7fae74159500 2023-10-06T01:42:30.583 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.580+0000 7fae7c01d700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae74145980 con 0x7fae740b3fd0 2023-10-06T01:42:30.583 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.580+0000 7fae7c01d700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae741537a0 con 0x7fae7415dab0 2023-10-06T01:42:30.583 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.580+0000 7fae795b8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae74159500 0x7fae74149210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:30.583 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.580+0000 7fae79db9700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fae740b3fd0 0x7fae74153fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:30.583 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.580+0000 7fae795b8700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae74159500 0x7fae74149210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48300/0 (socket says 172.21.15.27:48300) 2023-10-06T01:42:30.583 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.580+0000 7fae795b8700 1 -- 172.21.15.27:0/3193364635 learned_addr learned my addr 172.21.15.27:0/3193364635 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:30.583 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.580+0000 7fae79db9700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fae740b3fd0 0x7fae74153fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.27:41482/0 (socket says 172.21.15.27:41482) 2023-10-06T01:42:30.583 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.580+0000 7fae7a5ba700 1 --2- 172.21.15.27:0/3193364635 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fae7415dab0 0x7fae74143d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:30.583 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.580+0000 7fae795b8700 1 -- 172.21.15.27:0/3193364635 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fae7415dab0 msgr2=0x7fae74143d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:30.583 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.580+0000 7fae795b8700 1 --2- 172.21.15.27:0/3193364635 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fae7415dab0 0x7fae74143d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:30.584 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.580+0000 7fae795b8700 1 -- 172.21.15.27:0/3193364635 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fae740b3fd0 msgr2=0x7fae74153fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:30.584 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.580+0000 7fae795b8700 1 --2- 172.21.15.27:0/3193364635 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fae740b3fd0 0x7fae74153fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:30.584 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.580+0000 7fae795b8700 1 -- 172.21.15.27:0/3193364635 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae6c0094f0 con 0x7fae74159500 2023-10-06T01:42:30.584 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.580+0000 7fae795b8700 1 --2- 172.21.15.27:0/3193364635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae74159500 0x7fae74149210 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fae6800c590 tx=0x7fae6800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:30.584 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.580+0000 7fae71ffb700 1 -- 172.21.15.27:0/3193364635 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae68011590 con 0x7fae74159500 2023-10-06T01:42:30.584 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.580+0000 7fae71ffb700 1 -- 172.21.15.27:0/3193364635 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae68011710 con 0x7fae74159500 2023-10-06T01:42:30.584 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.580+0000 7fae7c01d700 1 -- 172.21.15.27:0/3193364635 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae74153a80 con 0x7fae74159500 2023-10-06T01:42:30.584 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.580+0000 7fae71ffb700 1 -- 172.21.15.27:0/3193364635 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae68011ad0 con 0x7fae74159500 2023-10-06T01:42:30.584 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.581+0000 7fae7c01d700 1 -- 172.21.15.27:0/3193364635 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae74154500 con 0x7fae74159500 2023-10-06T01:42:30.584 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.582+0000 7fae7c01d700 1 -- 172.21.15.27:0/3193364635 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae74064050 con 0x7fae74159500 2023-10-06T01:42:30.584 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.582+0000 7fae71ffb700 1 -- 172.21.15.27:0/3193364635 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fae6801b3b0 con 0x7fae74159500 2023-10-06T01:42:30.584 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.582+0000 7fae71ffb700 1 --2- 172.21.15.27:0/3193364635 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fae60066f60 0x7fae60069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:30.585 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.582+0000 7fae71ffb700 1 -- 172.21.15.27:0/3193364635 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5292+0+0 (secure 0 0 0) 0x7fae68083880 con 0x7fae74159500 2023-10-06T01:42:30.585 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.583+0000 7fae79db9700 1 --2- 172.21.15.27:0/3193364635 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fae60066f60 0x7fae60069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:30.585 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.583+0000 7fae79db9700 1 --2- 172.21.15.27:0/3193364635 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fae60066f60 0x7fae60069410 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fae6400f570 tx=0x7fae6400b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:30.587 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.587+0000 7fae71ffb700 1 -- 172.21.15.27:0/3193364635 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fae68050c90 con 0x7fae74159500 2023-10-06T01:42:30.782 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.782+0000 7fae7c01d700 1 -- 172.21.15.27:0/3193364635 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mds", "name": "mds_symlink_recovery"} v 0) v1 -- 0x7fae74145ac0 con 0x7fae74159500 2023-10-06T01:42:30.786 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.786+0000 7fae71ffb700 1 -- 172.21.15.27:0/3193364635 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mds", "name": "mds_symlink_recovery"}]=0 v13) v1 ==== 103+0+0 (secure 0 0 0) 0x7fae68017070 con 0x7fae74159500 2023-10-06T01:42:30.788 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.788+0000 7fae7c01d700 1 -- 172.21.15.27:0/3193364635 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fae60066f60 msgr2=0x7fae60069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:30.788 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.788+0000 7fae7c01d700 1 --2- 172.21.15.27:0/3193364635 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fae60066f60 0x7fae60069410 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fae6400f570 tx=0x7fae6400b040 comp rx=0 tx=0).stop 2023-10-06T01:42:30.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.788+0000 7fae7c01d700 1 -- 172.21.15.27:0/3193364635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae74159500 msgr2=0x7fae74149210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:30.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.788+0000 7fae7c01d700 1 --2- 172.21.15.27:0/3193364635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae74159500 0x7fae74149210 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fae6800c590 tx=0x7fae6800c8a0 comp rx=0 tx=0).stop 2023-10-06T01:42:30.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.788+0000 7fae7c01d700 1 -- 172.21.15.27:0/3193364635 shutdown_connections 2023-10-06T01:42:30.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.788+0000 7fae7c01d700 1 --2- 172.21.15.27:0/3193364635 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fae60066f60 0x7fae60069410 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:30.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.788+0000 7fae7c01d700 1 --2- 172.21.15.27:0/3193364635 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fae7415dab0 0x7fae74143d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:30.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.788+0000 7fae7c01d700 1 --2- 172.21.15.27:0/3193364635 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fae740b3fd0 0x7fae74153fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:30.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.788+0000 7fae7c01d700 1 --2- 172.21.15.27:0/3193364635 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fae74159500 0x7fae74149210 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:30.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.788+0000 7fae7c01d700 1 -- 172.21.15.27:0/3193364635 >> 172.21.15.27:0/3193364635 conn(0x7fae74067630 msgr2=0x7fae740cf860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:30.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.788+0000 7fae7c01d700 1 -- 172.21.15.27:0/3193364635 shutdown_connections 2023-10-06T01:42:30.789 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.788+0000 7fae7c01d700 1 -- 172.21.15.27:0/3193364635 wait complete. 2023-10-06T01:42:30.800 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-06T01:42:30.921 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.920+0000 7f93e3c38700 1 Processor -- start 2023-10-06T01:42:30.921 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.921+0000 7f93e3c38700 1 -- start start 2023-10-06T01:42:30.923 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.921+0000 7f93e3c38700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f93dc0b4000 0x7f93dc159b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:30.923 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.921+0000 7f93e3c38700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93dc0b1690 0x7f93dc0b1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:30.924 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.921+0000 7f93e3c38700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93dc0698e0 con 0x7f93dc15cb10 2023-10-06T01:42:30.924 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.921+0000 7f93e3c38700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93dc0b2020 con 0x7f93dc0b4000 2023-10-06T01:42:30.924 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.921+0000 7f93e3c38700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93dc068fd0 con 0x7f93dc0b1690 2023-10-06T01:42:30.924 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.922+0000 7f93e19d4700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f93dc15cb10 0x7f93dc0af780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51268/0 (socket says 172.21.15.27:51268) 2023-10-06T01:42:30.924 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.922+0000 7f93e19d4700 1 -- 172.21.15.27:0/3733503712 learned_addr learned my addr 172.21.15.27:0/3733503712 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:30.924 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.922+0000 7f93e21d5700 1 --2- 172.21.15.27:0/3733503712 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93dc0b1690 0x7f93dc0b1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:30.924 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.922+0000 7f93e11d3700 1 --2- 172.21.15.27:0/3733503712 >> v2:172.21.15.124:3300/0 conn(0x7f93dc0b4000 0x7f93dc159b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:30.924 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.922+0000 7f93e09d2700 1 -- 172.21.15.27:0/3733503712 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1592328771 0 0) 0x7f93dc0698e0 con 0x7f93dc15cb10 2023-10-06T01:42:30.924 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.922+0000 7f93e09d2700 1 -- 172.21.15.27:0/3733503712 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93c4003570 con 0x7f93dc15cb10 2023-10-06T01:42:30.924 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.922+0000 7f93e09d2700 1 -- 172.21.15.27:0/3733503712 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2214696543 0 0) 0x7f93c4003570 con 0x7f93dc15cb10 2023-10-06T01:42:30.924 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.922+0000 7f93e21d5700 1 -- 172.21.15.27:0/3733503712 >> v2:172.21.15.124:3300/0 conn(0x7f93dc0b4000 msgr2=0x7f93dc159b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:30.925 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.922+0000 7f93e21d5700 1 --2- 172.21.15.27:0/3733503712 >> v2:172.21.15.124:3300/0 conn(0x7f93dc0b4000 0x7f93dc159b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:30.925 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.922+0000 7f93e21d5700 1 -- 172.21.15.27:0/3733503712 >> v1:172.21.15.27:6789/0 conn(0x7f93dc15cb10 legacy=0x7f93dc0af780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:30.925 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.922+0000 7f93e21d5700 1 -- 172.21.15.27:0/3733503712 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93dc0b0020 con 0x7f93dc0b1690 2023-10-06T01:42:30.925 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.922+0000 7f93e11d3700 1 --2- 172.21.15.27:0/3733503712 >> v2:172.21.15.124:3300/0 conn(0x7f93dc0b4000 0x7f93dc159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:42:30.925 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.922+0000 7f93e21d5700 1 --2- 172.21.15.27:0/3733503712 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93dc0b1690 0x7f93dc0b1ae0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f93cc009e60 tx=0x7f93cc009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e78cc02cf50a53a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:30.925 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.923+0000 7f93e09d2700 1 -- 172.21.15.27:0/3733503712 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93cc00e070 con 0x7f93dc0b1690 2023-10-06T01:42:30.925 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.923+0000 7f93e09d2700 1 -- 172.21.15.27:0/3733503712 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93cc00b7a0 con 0x7f93dc0b1690 2023-10-06T01:42:30.925 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.923+0000 7f93e09d2700 1 -- 172.21.15.27:0/3733503712 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d4001470 con 0x7f93dc0b1690 2023-10-06T01:42:30.925 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.923+0000 7f93e3c38700 1 -- 172.21.15.27:0/3733503712 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93dc0b1690 msgr2=0x7f93dc0b1ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:30.926 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.923+0000 7f93e3c38700 1 --2- 172.21.15.27:0/3733503712 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93dc0b1690 0x7f93dc0b1ae0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f93cc009e60 tx=0x7f93cc009780 comp rx=0 tx=0).stop 2023-10-06T01:42:30.926 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.923+0000 7f93e3c38700 1 -- 172.21.15.27:0/3733503712 shutdown_connections 2023-10-06T01:42:30.926 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.923+0000 7f93e3c38700 1 --2- 172.21.15.27:0/3733503712 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93dc0b1690 0x7f93dc0b1ae0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:30.926 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.923+0000 7f93e3c38700 1 --2- 172.21.15.27:0/3733503712 >> v2:172.21.15.124:3300/0 conn(0x7f93dc0b4000 0x7f93dc159b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:30.926 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.923+0000 7f93e3c38700 1 -- 172.21.15.27:0/3733503712 >> 172.21.15.27:0/3733503712 conn(0x7f93dc06d890 msgr2=0x7f93dc06dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:30.926 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.923+0000 7f93e3c38700 1 -- 172.21.15.27:0/3733503712 shutdown_connections 2023-10-06T01:42:30.926 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.923+0000 7f93e3c38700 1 -- 172.21.15.27:0/3733503712 wait complete. 2023-10-06T01:42:30.929 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.924+0000 7f93e3c38700 1 Processor -- start 2023-10-06T01:42:30.929 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.924+0000 7f93e3c38700 1 -- start start 2023-10-06T01:42:30.929 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.925+0000 7f93e3c38700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93dc0b1690 0x7f93dc1586c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:30.929 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.925+0000 7f93e3c38700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93dc0b4000 0x7f93dc0a5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:30.929 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.925+0000 7f93e3c38700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f93dc15cb10 0x7f93dc0a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:30.929 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.925+0000 7f93e3c38700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93dc0cb2a0 con 0x7f93dc0b4000 2023-10-06T01:42:30.929 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.925+0000 7f93e3c38700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93dc0cc510 con 0x7f93dc15cb10 2023-10-06T01:42:30.930 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.925+0000 7f93e3c38700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93dc0cbf00 con 0x7f93dc0b1690 2023-10-06T01:42:30.930 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.925+0000 7f93e11d3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93dc0b4000 0x7f93dc0a5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:30.930 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.925+0000 7f93e11d3700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93dc0b4000 0x7f93dc0a5df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48308/0 (socket says 172.21.15.27:48308) 2023-10-06T01:42:30.930 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.925+0000 7f93e11d3700 1 -- 172.21.15.27:0/3529685801 learned_addr learned my addr 172.21.15.27:0/3529685801 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:30.930 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.925+0000 7f93e19d4700 1 --2- 172.21.15.27:0/3529685801 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93dc0b1690 0x7f93dc1586c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:30.930 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.925+0000 7f93e21d5700 1 --2- 172.21.15.27:0/3529685801 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f93dc15cb10 0x7f93dc0a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:30.930 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.926+0000 7f93e11d3700 1 -- 172.21.15.27:0/3529685801 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93dc0b1690 msgr2=0x7f93dc1586c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:30.930 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.926+0000 7f93e11d3700 1 --2- 172.21.15.27:0/3529685801 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93dc0b1690 0x7f93dc1586c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:30.930 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.926+0000 7f93e11d3700 1 -- 172.21.15.27:0/3529685801 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f93dc15cb10 msgr2=0x7f93dc0a6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:30.930 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.926+0000 7f93e11d3700 1 --2- 172.21.15.27:0/3529685801 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f93dc15cb10 0x7f93dc0a6330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:30.930 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.926+0000 7f93e11d3700 1 -- 172.21.15.27:0/3529685801 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d0009420 con 0x7f93dc0b4000 2023-10-06T01:42:30.930 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.926+0000 7f93e21d5700 1 --2- 172.21.15.27:0/3529685801 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f93dc15cb10 0x7f93dc0a6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:42:30.931 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.926+0000 7f93e11d3700 1 --2- 172.21.15.27:0/3529685801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93dc0b4000 0x7f93dc0a5df0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f93d000ec80 tx=0x7f93d00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:30.931 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.926+0000 7f93d9ffb700 1 -- 172.21.15.27:0/3529685801 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d000c9d0 con 0x7f93dc0b4000 2023-10-06T01:42:30.931 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.926+0000 7f93d9ffb700 1 -- 172.21.15.27:0/3529685801 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93d000cb30 con 0x7f93dc0b4000 2023-10-06T01:42:30.931 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.926+0000 7f93e3c38700 1 -- 172.21.15.27:0/3529685801 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93cc009a20 con 0x7f93dc0b4000 2023-10-06T01:42:30.931 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.926+0000 7f93d9ffb700 1 -- 172.21.15.27:0/3529685801 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d000ceb0 con 0x7f93dc0b4000 2023-10-06T01:42:30.931 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.926+0000 7f93e3c38700 1 -- 172.21.15.27:0/3529685801 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93dc0cc1a0 con 0x7f93dc0b4000 2023-10-06T01:42:30.931 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.927+0000 7f93e3c38700 1 -- 172.21.15.27:0/3529685801 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93dc0cbd60 con 0x7f93dc0b4000 2023-10-06T01:42:30.931 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.927+0000 7f93d9ffb700 1 -- 172.21.15.27:0/3529685801 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f93d0013680 con 0x7f93dc0b4000 2023-10-06T01:42:30.931 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.928+0000 7f93d9ffb700 1 --2- 172.21.15.27:0/3529685801 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f93c8066f30 0x7f93c80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:30.931 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.928+0000 7f93d9ffb700 1 -- 172.21.15.27:0/3529685801 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5292+0+0 (secure 0 0 0) 0x7f93d0084790 con 0x7f93dc0b4000 2023-10-06T01:42:30.931 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.928+0000 7f93e19d4700 1 --2- 172.21.15.27:0/3529685801 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f93c8066f30 0x7f93c80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:30.931 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.929+0000 7f93e19d4700 1 --2- 172.21.15.27:0/3529685801 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f93c8066f30 0x7f93c80693e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f93d4001ed0 tx=0x7f93d4001520 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:30.932 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:30.932+0000 7f93d9ffb700 1 -- 172.21.15.27:0/3529685801 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f93dc0cbd60 con 0x7f93dc0b4000 2023-10-06T01:42:31.129 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.129+0000 7f93e3c38700 1 -- 172.21.15.27:0/3529685801 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f93dc0cbd60 con 0x7f93dc0b4000 2023-10-06T01:42:31.133 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.133+0000 7f93d9ffb700 1 -- 172.21.15.27:0/3529685801 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v14) v1 ==== 104+0+0 (secure 0 0 0) 0x7f93dc0cbd60 con 0x7f93dc0b4000 2023-10-06T01:42:31.135 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.134+0000 7f93e3c38700 1 -- 172.21.15.27:0/3529685801 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f93c8066f30 msgr2=0x7f93c80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:31.135 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.134+0000 7f93e3c38700 1 --2- 172.21.15.27:0/3529685801 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f93c8066f30 0x7f93c80693e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f93d4001ed0 tx=0x7f93d4001520 comp rx=0 tx=0).stop 2023-10-06T01:42:31.135 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.135+0000 7f93e3c38700 1 -- 172.21.15.27:0/3529685801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93dc0b4000 msgr2=0x7f93dc0a5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:31.135 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.135+0000 7f93e3c38700 1 --2- 172.21.15.27:0/3529685801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93dc0b4000 0x7f93dc0a5df0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f93d000ec80 tx=0x7f93d00091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:31.135 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.135+0000 7f93e3c38700 1 -- 172.21.15.27:0/3529685801 shutdown_connections 2023-10-06T01:42:31.136 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.135+0000 7f93e3c38700 1 --2- 172.21.15.27:0/3529685801 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f93c8066f30 0x7f93c80693e0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:31.136 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.135+0000 7f93e3c38700 1 --2- 172.21.15.27:0/3529685801 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f93dc0b1690 0x7f93dc1586c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:31.136 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.135+0000 7f93e3c38700 1 --2- 172.21.15.27:0/3529685801 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f93dc15cb10 0x7f93dc0a6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:31.136 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.135+0000 7f93e3c38700 1 --2- 172.21.15.27:0/3529685801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f93dc0b4000 0x7f93dc0a5df0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:31.136 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.135+0000 7f93e3c38700 1 -- 172.21.15.27:0/3529685801 >> 172.21.15.27:0/3529685801 conn(0x7f93dc06d890 msgr2=0x7f93dc15e710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:31.136 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.135+0000 7f93e3c38700 1 -- 172.21.15.27:0/3529685801 shutdown_connections 2023-10-06T01:42:31.136 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.135+0000 7f93e3c38700 1 -- 172.21.15.27:0/3529685801 wait complete. 2023-10-06T01:42:31.147 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_backtrace.TestBacktrace.test_backtrace' 2023-10-06T01:42:31.266 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.266+0000 7efdb7738700 1 Processor -- start 2023-10-06T01:42:31.266 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.266+0000 7efdb7738700 1 -- start start 2023-10-06T01:42:31.267 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.267+0000 7efdb7738700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7efdb015d930 0x7efdb00d0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:31.267 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.267+0000 7efdb7738700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7efdb01591b0 0x7efdb0159630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:31.267 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.267+0000 7efdb7738700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdb00b2ad0 con 0x7efdb015d560 2023-10-06T01:42:31.267 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.267+0000 7efdb7738700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdb00b4160 con 0x7efdb015d930 2023-10-06T01:42:31.267 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.267+0000 7efdb7738700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdb0159b70 con 0x7efdb01591b0 2023-10-06T01:42:31.269 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.267+0000 7efdb5cd5700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7efdb01591b0 0x7efdb0159630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:31.269 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.267+0000 7efdb4cd3700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7efdb015d930 0x7efdb00d0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:31.269 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.267+0000 7efdb5cd5700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7efdb01591b0 0x7efdb0159630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.27:45802/0 (socket says 172.21.15.27:45802) 2023-10-06T01:42:31.269 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.267+0000 7efdb5cd5700 1 -- 172.21.15.27:0/3562961431 learned_addr learned my addr 172.21.15.27:0/3562961431 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:31.269 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.267+0000 7efdb5cd5700 1 -- 172.21.15.27:0/3562961431 >> v2:172.21.15.124:3300/0 conn(0x7efdb015d930 msgr2=0x7efdb00d0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:31.269 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.267+0000 7efdb5cd5700 1 --2- 172.21.15.27:0/3562961431 >> v2:172.21.15.124:3300/0 conn(0x7efdb015d930 0x7efdb00d0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:31.269 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.267+0000 7efdb5cd5700 1 -- 172.21.15.27:0/3562961431 >> v1:172.21.15.124:6789/0 conn(0x7efdb015d560 legacy=0x7efdb00b1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:31.269 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.267+0000 7efdb5cd5700 1 -- 172.21.15.27:0/3562961431 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb00b2230 con 0x7efdb01591b0 2023-10-06T01:42:31.270 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.267+0000 7efdb4cd3700 1 --2- 172.21.15.27:0/3562961431 >> v2:172.21.15.124:3300/0 conn(0x7efdb015d930 0x7efdb00d0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:42:31.270 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.268+0000 7efdb5cd5700 1 --2- 172.21.15.27:0/3562961431 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7efdb01591b0 0x7efdb0159630 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efda0009f20 tx=0x7efda0009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca67043b38f4ca28 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:31.270 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.268+0000 7efdaffff700 1 -- 172.21.15.27:0/3562961431 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda000e070 con 0x7efdb01591b0 2023-10-06T01:42:31.270 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.268+0000 7efdaffff700 1 -- 172.21.15.27:0/3562961431 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efda000b9d0 con 0x7efdb01591b0 2023-10-06T01:42:31.270 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.268+0000 7efdaffff700 1 -- 172.21.15.27:0/3562961431 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda00092e0 con 0x7efdb01591b0 2023-10-06T01:42:31.270 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.268+0000 7efdb7738700 1 -- 172.21.15.27:0/3562961431 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7efdb01591b0 msgr2=0x7efdb0159630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:31.270 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.268+0000 7efdb7738700 1 --2- 172.21.15.27:0/3562961431 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7efdb01591b0 0x7efdb0159630 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efda0009f20 tx=0x7efda0009840 comp rx=0 tx=0).stop 2023-10-06T01:42:31.270 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.268+0000 7efdb7738700 1 -- 172.21.15.27:0/3562961431 shutdown_connections 2023-10-06T01:42:31.270 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.268+0000 7efdb7738700 1 --2- 172.21.15.27:0/3562961431 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7efdb01591b0 0x7efdb0159630 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:31.270 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.268+0000 7efdb7738700 1 --2- 172.21.15.27:0/3562961431 >> v2:172.21.15.124:3300/0 conn(0x7efdb015d930 0x7efdb00d0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:31.270 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.268+0000 7efdb7738700 1 -- 172.21.15.27:0/3562961431 >> 172.21.15.27:0/3562961431 conn(0x7efdb00d1180 msgr2=0x7efdb0069640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:31.270 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.269+0000 7efdb7738700 1 -- 172.21.15.27:0/3562961431 shutdown_connections 2023-10-06T01:42:31.271 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.269+0000 7efdb7738700 1 -- 172.21.15.27:0/3562961431 wait complete. 2023-10-06T01:42:31.273 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.270+0000 7efdb7738700 1 Processor -- start 2023-10-06T01:42:31.273 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.270+0000 7efdb7738700 1 -- start start 2023-10-06T01:42:31.273 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.270+0000 7efdb7738700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdb01591b0 0x7efdb00acdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:31.273 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.270+0000 7efdb7738700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7efdb015d560 0x7efdb0155720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:31.273 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.270+0000 7efdb7738700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7efdb015d930 0x7efdb015b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:31.273 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.270+0000 7efdb7738700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb00ab8a0 con 0x7efdb01591b0 2023-10-06T01:42:31.273 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.270+0000 7efdb7738700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb00a88c0 con 0x7efdb015d560 2023-10-06T01:42:31.273 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.270+0000 7efdb7738700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdb00ab2e0 con 0x7efdb015d930 2023-10-06T01:42:31.273 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.271+0000 7efdb54d4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdb01591b0 0x7efdb00acdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:31.273 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.271+0000 7efdb5cd5700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7efdb015d930 0x7efdb015b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:31.273 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.271+0000 7efdb4cd3700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7efdb015d560 0x7efdb0155720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:31.273 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.271+0000 7efdb54d4700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdb01591b0 0x7efdb00acdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48320/0 (socket says 172.21.15.27:48320) 2023-10-06T01:42:31.274 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.271+0000 7efdb54d4700 1 -- 172.21.15.27:0/4186619220 learned_addr learned my addr 172.21.15.27:0/4186619220 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:31.274 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.271+0000 7efdb54d4700 1 -- 172.21.15.27:0/4186619220 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7efdb015d930 msgr2=0x7efdb015b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:31.274 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.271+0000 7efdb54d4700 1 --2- 172.21.15.27:0/4186619220 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7efdb015d930 0x7efdb015b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:31.274 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.271+0000 7efdb54d4700 1 -- 172.21.15.27:0/4186619220 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7efdb015d560 msgr2=0x7efdb0155720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:31.274 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.271+0000 7efdb54d4700 1 --2- 172.21.15.27:0/4186619220 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7efdb015d560 0x7efdb0155720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:31.274 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.271+0000 7efdb54d4700 1 -- 172.21.15.27:0/4186619220 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda4009420 con 0x7efdb01591b0 2023-10-06T01:42:31.274 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.271+0000 7efdb4cd3700 1 --2- 172.21.15.27:0/4186619220 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7efdb015d560 0x7efdb0155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:42:31.274 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.271+0000 7efdb54d4700 1 --2- 172.21.15.27:0/4186619220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdb01591b0 0x7efdb00acdc0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7efda800c590 tx=0x7efda800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:31.274 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.271+0000 7efdad7fa700 1 -- 172.21.15.27:0/4186619220 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda8011590 con 0x7efdb01591b0 2023-10-06T01:42:31.274 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.271+0000 7efdad7fa700 1 -- 172.21.15.27:0/4186619220 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efda8011710 con 0x7efdb01591b0 2023-10-06T01:42:31.274 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.271+0000 7efdb7738700 1 -- 172.21.15.27:0/4186619220 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efda0009ae0 con 0x7efdb01591b0 2023-10-06T01:42:31.274 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.271+0000 7efdad7fa700 1 -- 172.21.15.27:0/4186619220 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efda8011ad0 con 0x7efdb01591b0 2023-10-06T01:42:31.275 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.271+0000 7efdb7738700 1 -- 172.21.15.27:0/4186619220 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdb00a8550 con 0x7efdb01591b0 2023-10-06T01:42:31.275 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.272+0000 7efdb7738700 1 -- 172.21.15.27:0/4186619220 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdb0064050 con 0x7efdb01591b0 2023-10-06T01:42:31.275 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.273+0000 7efdad7fa700 1 -- 172.21.15.27:0/4186619220 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7efda801b3b0 con 0x7efdb01591b0 2023-10-06T01:42:31.275 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.273+0000 7efdad7fa700 1 --2- 172.21.15.27:0/4186619220 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7efd9c066f30 0x7efd9c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:31.275 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.273+0000 7efdad7fa700 1 -- 172.21.15.27:0/4186619220 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5292+0+0 (secure 0 0 0) 0x7efda8083740 con 0x7efdb01591b0 2023-10-06T01:42:31.276 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.273+0000 7efdb4cd3700 1 --2- 172.21.15.27:0/4186619220 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7efd9c066f30 0x7efd9c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:31.276 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.274+0000 7efdb4cd3700 1 --2- 172.21.15.27:0/4186619220 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7efd9c066f30 0x7efd9c0693e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7efdb00b43d0 tx=0x7efda40091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:31.277 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.277+0000 7efdad7fa700 1 -- 172.21.15.27:0/4186619220 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efda8050b50 con 0x7efdb01591b0 2023-10-06T01:42:31.549 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:31.549+0000 7efdb7738700 1 -- 172.21.15.27:0/4186619220 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_backtrace.TestBacktrace.test_backtrace"]} v 0) v1 -- 0x7efdb00b2510 con 0x7efdb01591b0 2023-10-06T01:42:32.441 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:32.441+0000 7efdad7fa700 1 -- 172.21.15.27:0/4186619220 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_backtrace.TestBacktrace.test_backtrace"]}]=0 v93) v1 ==== 135+0+0 (secure 0 0 0) 0x7efda8017020 con 0x7efdb01591b0 2023-10-06T01:42:32.443 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:32.443+0000 7efdb7738700 1 -- 172.21.15.27:0/4186619220 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7efd9c066f30 msgr2=0x7efd9c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:32.444 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:32.443+0000 7efdb7738700 1 --2- 172.21.15.27:0/4186619220 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7efd9c066f30 0x7efd9c0693e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7efdb00b43d0 tx=0x7efda40091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:32.444 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:32.443+0000 7efdb7738700 1 -- 172.21.15.27:0/4186619220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdb01591b0 msgr2=0x7efdb00acdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:32.444 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:32.443+0000 7efdb7738700 1 --2- 172.21.15.27:0/4186619220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdb01591b0 0x7efdb00acdc0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7efda800c590 tx=0x7efda800c8a0 comp rx=0 tx=0).stop 2023-10-06T01:42:32.445 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:32.443+0000 7efdb7738700 1 -- 172.21.15.27:0/4186619220 shutdown_connections 2023-10-06T01:42:32.445 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:32.443+0000 7efdb7738700 1 --2- 172.21.15.27:0/4186619220 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7efd9c066f30 0x7efd9c0693e0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:32.446 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:32.443+0000 7efdb7738700 1 --2- 172.21.15.27:0/4186619220 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7efdb015d930 0x7efdb015b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:32.446 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:32.443+0000 7efdb7738700 1 --2- 172.21.15.27:0/4186619220 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7efdb01591b0 0x7efdb00acdc0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:32.446 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:32.443+0000 7efdb7738700 1 --2- 172.21.15.27:0/4186619220 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7efdb015d560 0x7efdb0155720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:32.446 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:32.443+0000 7efdb7738700 1 -- 172.21.15.27:0/4186619220 >> 172.21.15.27:0/4186619220 conn(0x7efdb00d1180 msgr2=0x7efdb0156b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:32.447 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:32.443+0000 7efdb7738700 1 -- 172.21.15.27:0/4186619220 shutdown_connections 2023-10-06T01:42:32.447 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:32.443+0000 7efdb7738700 1 -- 172.21.15.27:0/4186619220 wait complete. 2023-10-06T01:42:32.457 INFO:tasks.cephfs_test_runner:That the 'parent' 'layout' and 'symlink' xattrs on the head objects of files ... ok 2023-10-06T01:42:32.458 INFO:tasks.cephfs_test_runner: 2023-10-06T01:42:32.458 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-10-06T01:42:32.458 INFO:tasks.cephfs_test_runner:Ran 1 test in 46.341s 2023-10-06T01:42:32.458 INFO:tasks.cephfs_test_runner: 2023-10-06T01:42:32.458 INFO:tasks.cephfs_test_runner:OK 2023-10-06T01:42:32.458 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2023-10-06T01:42:32.465 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-10-06T01:42:32.473 INFO:tasks.kclient:Unmounting kernel clients... 2023-10-06T01:42:32.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:42:32.473 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2023-10-06T01:42:32.489 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-06T01:42:32.489 INFO:teuthology.orchestra.run.smithi027.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-06T01:42:32.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:42:32.489 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-06T01:42:32.566 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:42:32.567 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-06T01:42:32.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:42:32.643 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link delete ceph-brx 2023-10-06T01:42:32.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:42:32.749 DEBUG:teuthology.orchestra.run.smithi027:> ip netns list 2023-10-06T01:42:32.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:42:32.765 DEBUG:teuthology.orchestra.run.smithi027:> sudo ip link delete ceph-brx 2023-10-06T01:42:32.834 INFO:teuthology.orchestra.run.smithi027.stderr:Cannot find device "ceph-brx" 2023-10-06T01:42:32.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T01:42:32.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:42:32.835 DEBUG:teuthology.orchestra.run.smithi124:> ip netns list 2023-10-06T01:42:32.853 INFO:teuthology.orchestra.run.smithi124.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-10-06T01:42:32.853 INFO:teuthology.orchestra.run.smithi124.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-10-06T01:42:32.854 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:42:32.854 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2023-10-06T01:42:32.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:42:32.935 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-06T01:42:33.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:42:33.023 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link delete ceph-brx 2023-10-06T01:42:33.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:42:33.103 DEBUG:teuthology.orchestra.run.smithi124:> ip netns list 2023-10-06T01:42:33.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-06T01:42:33.119 DEBUG:teuthology.orchestra.run.smithi124:> sudo ip link delete ceph-brx 2023-10-06T01:42:33.187 INFO:teuthology.orchestra.run.smithi124.stderr:Cannot find device "ceph-brx" 2023-10-06T01:42:33.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T01:42:33.189 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-06T01:42:33.199 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-10-06T01:42:33.199 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T01:42:33.323 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.322+0000 7f725359e700 1 Processor -- start 2023-10-06T01:42:33.323 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.322+0000 7f725359e700 1 -- start start 2023-10-06T01:42:33.324 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.323+0000 7f725359e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f725415d480 0x7f7254069860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:33.324 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.323+0000 7f725359e700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f7254069da0 0x7f72540b0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.323+0000 7f725359e700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72540687c0 con 0x7f725415d850 2023-10-06T01:42:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.323+0000 7f725359e700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7254068aa0 con 0x7f7254069da0 2023-10-06T01:42:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.323+0000 7f725359e700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7254068d80 con 0x7f725415d480 2023-10-06T01:42:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.324+0000 7f7252d9d700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f725415d850 0x7f7254142a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51282/0 (socket says 172.21.15.27:51282) 2023-10-06T01:42:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.324+0000 7f7252d9d700 1 -- 172.21.15.27:0/3951771676 learned_addr learned my addr 172.21.15.27:0/3951771676 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.324+0000 7f725259c700 1 --2- 172.21.15.27:0/3951771676 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f725415d480 0x7f7254069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.324+0000 7f7251d9b700 1 --2- 172.21.15.27:0/3951771676 >> v2:172.21.15.124:3300/0 conn(0x7f7254069da0 0x7f72540b0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.324+0000 7f725159a700 1 -- 172.21.15.27:0/3951771676 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1705927089 0 0) 0x7f72540687c0 con 0x7f725415d850 2023-10-06T01:42:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.324+0000 7f725159a700 1 -- 172.21.15.27:0/3951771676 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7240003570 con 0x7f725415d850 2023-10-06T01:42:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.324+0000 7f725259c700 1 -- 172.21.15.27:0/3951771676 >> v2:172.21.15.124:3300/0 conn(0x7f7254069da0 msgr2=0x7f72540b0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.324+0000 7f725259c700 1 --2- 172.21.15.27:0/3951771676 >> v2:172.21.15.124:3300/0 conn(0x7f7254069da0 0x7f72540b0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:33.325 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.324+0000 7f725259c700 1 -- 172.21.15.27:0/3951771676 >> v1:172.21.15.27:6789/0 conn(0x7f725415d850 legacy=0x7f7254142a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:33.326 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.324+0000 7f725259c700 1 -- 172.21.15.27:0/3951771676 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7254069060 con 0x7f725415d480 2023-10-06T01:42:33.326 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.324+0000 7f725259c700 1 --2- 172.21.15.27:0/3951771676 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f725415d480 0x7f7254069860 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f724c0098e0 tx=0x7f724c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dab82309732b6e16 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:33.326 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.324+0000 7f725159a700 1 -- 172.21.15.27:0/3951771676 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724c011630 con 0x7f725415d480 2023-10-06T01:42:33.326 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.325+0000 7f725159a700 1 -- 172.21.15.27:0/3951771676 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f724c011790 con 0x7f725415d480 2023-10-06T01:42:33.326 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.325+0000 7f725159a700 1 -- 172.21.15.27:0/3951771676 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724c011b30 con 0x7f725415d480 2023-10-06T01:42:33.328 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.325+0000 7f725359e700 1 -- 172.21.15.27:0/3951771676 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f725415d480 msgr2=0x7f7254069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:33.328 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.325+0000 7f725359e700 1 --2- 172.21.15.27:0/3951771676 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f725415d480 0x7f7254069860 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f724c0098e0 tx=0x7f724c0091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:33.329 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.325+0000 7f725359e700 1 -- 172.21.15.27:0/3951771676 shutdown_connections 2023-10-06T01:42:33.329 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.325+0000 7f725359e700 1 --2- 172.21.15.27:0/3951771676 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f725415d480 0x7f7254069860 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:33.329 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.325+0000 7f725359e700 1 --2- 172.21.15.27:0/3951771676 >> v2:172.21.15.124:3300/0 conn(0x7f7254069da0 0x7f72540b0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:33.329 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.325+0000 7f725359e700 1 -- 172.21.15.27:0/3951771676 >> 172.21.15.27:0/3951771676 conn(0x7f72540ccf80 msgr2=0x7f72540cd380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:33.329 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.325+0000 7f725359e700 1 -- 172.21.15.27:0/3951771676 shutdown_connections 2023-10-06T01:42:33.329 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.325+0000 7f725359e700 1 -- 172.21.15.27:0/3951771676 wait complete. 2023-10-06T01:42:33.329 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.326+0000 7f725359e700 1 Processor -- start 2023-10-06T01:42:33.329 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.326+0000 7f725359e700 1 -- start start 2023-10-06T01:42:33.329 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.327+0000 7f725359e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7254069da0 0x7f72540b0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:33.329 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.327+0000 7f725359e700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f725415d480 0x7f72540aeee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:33.329 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.327+0000 7f725359e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f725415d850 0x7f72540af420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:33.330 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.327+0000 7f725359e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72540c9760 con 0x7f725415d850 2023-10-06T01:42:33.330 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.327+0000 7f725359e700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7254132c00 con 0x7f725415d480 2023-10-06T01:42:33.330 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.327+0000 7f725359e700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f725416ddd0 con 0x7f7254069da0 2023-10-06T01:42:33.330 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.327+0000 7f7252d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f725415d850 0x7f72540af420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:33.330 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.327+0000 7f7252d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f725415d850 0x7f72540af420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48328/0 (socket says 172.21.15.27:48328) 2023-10-06T01:42:33.330 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.327+0000 7f7252d9d700 1 -- 172.21.15.27:0/324499002 learned_addr learned my addr 172.21.15.27:0/324499002 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:33.330 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.327+0000 7f725259c700 1 --2- 172.21.15.27:0/324499002 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7254069da0 0x7f72540b0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:33.330 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.327+0000 7f7251d9b700 1 --2- 172.21.15.27:0/324499002 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f725415d480 0x7f72540aeee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:33.330 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.327+0000 7f7252d9d700 1 -- 172.21.15.27:0/324499002 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7254069da0 msgr2=0x7f72540b0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:33.330 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.327+0000 7f7252d9d700 1 --2- 172.21.15.27:0/324499002 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7254069da0 0x7f72540b0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:33.330 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.327+0000 7f7252d9d700 1 -- 172.21.15.27:0/324499002 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f725415d480 msgr2=0x7f72540aeee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:33.330 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.327+0000 7f7252d9d700 1 --2- 172.21.15.27:0/324499002 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f725415d480 0x7f72540aeee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:33.331 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.327+0000 7f7252d9d700 1 -- 172.21.15.27:0/324499002 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f724c009420 con 0x7f725415d850 2023-10-06T01:42:33.331 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.327+0000 7f7251d9b700 1 --2- 172.21.15.27:0/324499002 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f725415d480 0x7f72540aeee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:42:33.331 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.327+0000 7f7252d9d700 1 --2- 172.21.15.27:0/324499002 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f725415d850 0x7f72540af420 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f7244001460 tx=0x7f7244001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:33.331 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.328+0000 7f724a7fc700 1 -- 172.21.15.27:0/324499002 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724400a8a0 con 0x7f725415d850 2023-10-06T01:42:33.331 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.328+0000 7f724a7fc700 1 -- 172.21.15.27:0/324499002 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f724400aa00 con 0x7f725415d850 2023-10-06T01:42:33.331 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.328+0000 7f725359e700 1 -- 172.21.15.27:0/324499002 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72540c8dc0 con 0x7f725415d850 2023-10-06T01:42:33.331 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.328+0000 7f724a7fc700 1 -- 172.21.15.27:0/324499002 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f724400ad80 con 0x7f725415d850 2023-10-06T01:42:33.331 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.328+0000 7f725359e700 1 -- 172.21.15.27:0/324499002 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72540c9380 con 0x7f725415d850 2023-10-06T01:42:33.332 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.329+0000 7f725359e700 1 -- 172.21.15.27:0/324499002 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72540c98a0 con 0x7f725415d850 2023-10-06T01:42:33.332 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.329+0000 7f724a7fc700 1 -- 172.21.15.27:0/324499002 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7244007300 con 0x7f725415d850 2023-10-06T01:42:33.332 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.329+0000 7f724a7fc700 1 --2- 172.21.15.27:0/324499002 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7234066f30 0x7f72340693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:33.332 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.330+0000 7f724a7fc700 1 -- 172.21.15.27:0/324499002 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5292+0+0 (secure 0 0 0) 0x7f7244084870 con 0x7f725415d850 2023-10-06T01:42:33.332 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.330+0000 7f725259c700 1 --2- 172.21.15.27:0/324499002 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7234066f30 0x7f72340693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:33.332 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.330+0000 7f725259c700 1 --2- 172.21.15.27:0/324499002 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7234066f30 0x7f72340693e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f724c0091e0 tx=0x7f724c017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:33.334 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.334+0000 7f724a7fc700 1 -- 172.21.15.27:0/324499002 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f72540c98a0 con 0x7f725415d850 2023-10-06T01:42:33.532 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.532+0000 7f725359e700 1 -- 172.21.15.27:0/324499002 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72540c98a0 con 0x7f725415d850 2023-10-06T01:42:33.533 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.533+0000 7f724a7fc700 1 -- 172.21.15.27:0/324499002 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+13779 (secure 0 0 0) 0x7f72540c98a0 con 0x7f725415d850 2023-10-06T01:42:33.533 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:33.533 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":38,"fsid":"20a7bff0-74f8-4f2f-b6c6-899e804bc965","created":"2023-10-06T01:40:54.474215+0000","modified":"2023-10-06T01:42:30.427970+0000","last_up_change":"2023-10-06T01:41:05.242847+0000","last_in_change":"2023-10-06T01:40:56.890960+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-10-06T01:41:05.685386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-10-06T01:41:06.092995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b902c86d-800c-40c5-b152-b545e50d02c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":90644},{"type":"v1","addr":"172.21.15.27:6801","nonce":90644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":90644},{"type":"v1","addr":"172.21.15.27:6803","nonce":90644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":90644},{"type":"v1","addr":"172.21.15.27:6807","nonce":90644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":90644},{"type":"v1","addr":"172.21.15.27:6805","nonce":90644}]},"public_addr":"172.21.15.27:6801/90644","cluster_addr":"172.21.15.27:6803/90644","heartbeat_back_addr":"172.21.15.27:6807/90644","heartbeat_front_addr":"172.21.15.27:6805/90644","state":["exists","up"]},{"osd":1,"uuid":"5b4a70e5-b06e-48cd-bfc4-7962c5d05386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":90643},{"type":"v1","addr":"172.21.15.27:6809","nonce":90643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":90643},{"type":"v1","addr":"172.21.15.27:6811","nonce":90643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":90643},{"type":"v1","addr":"172.21.15.27:6815","nonce":90643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":90643},{"type":"v1","addr":"172.21.15.27:6813","nonce":90643}]},"public_addr":"172.21.15.27:6809/90643","cluster_addr":"172.21.15.27:6811/90643","heartbeat_back_addr":"172.21.15.27:6815/90643","heartbeat_front_addr":"172.21.15.27:6813/90643","state":["exists","up"]},{"osd":2,"uuid":"22d9de46-4073-4336-9122-5227e601eca0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":90645},{"type":"v1","addr":"172.21.15.27:6817","nonce":90645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":90645},{"type":"v1","addr":"172.21.15.27:6819","nonce":90645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":90645},{"type":"v1","addr":"172.21.15.27:6823","nonce":90645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":90645},{"type":"v1","addr":"172.21.15.27:6821","nonce":90645}]},"public_addr":"172.21.15.27:6817/90645","cluster_addr":"172.21.15.27:6819/90645","heartbeat_back_addr":"172.21.15.27:6823/90645","heartbeat_front_addr":"172.21.15.27:6821/90645","state":["exists","up"]},{"osd":3,"uuid":"438bb2d3-34e2-4b7c-9734-8360c1b67650","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":90668},{"type":"v1","addr":"172.21.15.27:6825","nonce":90668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":90668},{"type":"v1","addr":"172.21.15.27:6827","nonce":90668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":90668},{"type":"v1","addr":"172.21.15.27:6831","nonce":90668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":90668},{"type":"v1","addr":"172.21.15.27:6829","nonce":90668}]},"public_addr":"172.21.15.27:6825/90668","cluster_addr":"172.21.15.27:6827/90668","heartbeat_back_addr":"172.21.15.27:6831/90668","heartbeat_front_addr":"172.21.15.27:6829/90668","state":["exists","up"]},{"osd":4,"uuid":"e1956a8e-e0f8-4e7c-acbb-40037415a354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":90863},{"type":"v1","addr":"172.21.15.124:6801","nonce":90863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":90863},{"type":"v1","addr":"172.21.15.124:6803","nonce":90863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":90863},{"type":"v1","addr":"172.21.15.124:6807","nonce":90863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":90863},{"type":"v1","addr":"172.21.15.124:6805","nonce":90863}]},"public_addr":"172.21.15.124:6801/90863","cluster_addr":"172.21.15.124:6803/90863","heartbeat_back_addr":"172.21.15.124:6807/90863","heartbeat_front_addr":"172.21.15.124:6805/90863","state":["exists","up"]},{"osd":5,"uuid":"306c59d4-846d-4b73-b1c9-051bec9e9368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":90885},{"type":"v1","addr":"172.21.15.124:6817","nonce":90885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":90885},{"type":"v1","addr":"172.21.15.124:6819","nonce":90885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":90885},{"type":"v1","addr":"172.21.15.124:6823","nonce":90885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":90885},{"type":"v1","addr":"172.21.15.124:6821","nonce":90885}]},"public_addr":"172.21.15.124:6817/90885","cluster_addr":"172.21.15.124:6819/90885","heartbeat_back_addr":"172.21.15.124:6823/90885","heartbeat_front_addr":"172.21.15.124:6821/90885","state":["exists","up"]},{"osd":6,"uuid":"15463e53-ac8d-47d1-b4f6-c0d31f29e4a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":90864},{"type":"v1","addr":"172.21.15.124:6809","nonce":90864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":90864},{"type":"v1","addr":"172.21.15.124:6811","nonce":90864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":90864},{"type":"v1","addr":"172.21.15.124:6815","nonce":90864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":90864},{"type":"v1","addr":"172.21.15.124:6813","nonce":90864}]},"public_addr":"172.21.15.124:6809/90864","cluster_addr":"172.21.15.124:6811/90864","heartbeat_back_addr":"172.21.15.124:6815/90864","heartbeat_front_addr":"172.21.15.124:6813/90864","state":["exists","up"]},{"osd":7,"uuid":"ae463a90-a9d3-498f-86fa-47a35b3fe889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":90894},{"type":"v1","addr":"172.21.15.124:6825","nonce":90894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":90894},{"type":"v1","addr":"172.21.15.124:6827","nonce":90894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":90894},{"type":"v1","addr":"172.21.15.124:6831","nonce":90894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":90894},{"type":"v1","addr":"172.21.15.124:6829","nonce":90894}]},"public_addr":"172.21.15.124:6825/90894","cluster_addr":"172.21.15.124:6827/90894","heartbeat_back_addr":"172.21.15.124:6831/90894","heartbeat_front_addr":"172.21.15.124:6829/90894","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.827704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.839951+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.032230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.050605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.987908+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.031290+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.080707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.053692+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.124:6837/325247112":"2023-10-07T01:42:26.110568+0000","172.21.15.124:6835/325247112":"2023-10-07T01:42:26.110568+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T01:42:33.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.535+0000 7f725359e700 1 -- 172.21.15.27:0/324499002 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7234066f30 msgr2=0x7f72340693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:33.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.535+0000 7f725359e700 1 --2- 172.21.15.27:0/324499002 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7234066f30 0x7f72340693e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f724c0091e0 tx=0x7f724c017000 comp rx=0 tx=0).stop 2023-10-06T01:42:33.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.535+0000 7f725359e700 1 -- 172.21.15.27:0/324499002 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f725415d850 msgr2=0x7f72540af420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:33.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.535+0000 7f725359e700 1 --2- 172.21.15.27:0/324499002 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f725415d850 0x7f72540af420 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f7244001460 tx=0x7f7244001a40 comp rx=0 tx=0).stop 2023-10-06T01:42:33.536 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.535+0000 7f725359e700 1 -- 172.21.15.27:0/324499002 shutdown_connections 2023-10-06T01:42:33.536 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.535+0000 7f725359e700 1 --2- 172.21.15.27:0/324499002 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7234066f30 0x7f72340693e0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:33.536 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.535+0000 7f725359e700 1 --2- 172.21.15.27:0/324499002 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7254069da0 0x7f72540b0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:33.536 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.535+0000 7f725359e700 1 --2- 172.21.15.27:0/324499002 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f725415d850 0x7f72540af420 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:33.536 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.535+0000 7f725359e700 1 --2- 172.21.15.27:0/324499002 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f725415d480 0x7f72540aeee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:33.536 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.535+0000 7f725359e700 1 -- 172.21.15.27:0/324499002 >> 172.21.15.27:0/324499002 conn(0x7f72540ccf80 msgr2=0x7f72540b11c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:33.536 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.535+0000 7f725359e700 1 -- 172.21.15.27:0/324499002 shutdown_connections 2023-10-06T01:42:33.536 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.535+0000 7f725359e700 1 -- 172.21.15.27:0/324499002 wait complete. 2023-10-06T01:42:33.547 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-06T01:42:33.547 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-06T01:42:33.667 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.667+0000 7fe75cc40700 1 Processor -- start 2023-10-06T01:42:33.667 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.667+0000 7fe75cc40700 1 -- start start 2023-10-06T01:42:33.669 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.668+0000 7fe75cc40700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe7580cdc00 0x7fe7580cbe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:33.669 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.668+0000 7fe75cc40700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe7580c6dc0 0x7fe7580ae5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:33.669 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.668+0000 7fe75cc40700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7580ae140 con 0x7fe7580d1030 2023-10-06T01:42:33.669 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.668+0000 7fe75cc40700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7580cc390 con 0x7fe7580c6dc0 2023-10-06T01:42:33.670 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.668+0000 7fe75cc40700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7580ad530 con 0x7fe7580cdc00 2023-10-06T01:42:33.670 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.668+0000 7fe756d9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe7580c6dc0 0x7fe7580ae5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:33.670 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.668+0000 7fe755d9b700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fe7580d1030 0x7fe7580c66b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51298/0 (socket says 172.21.15.27:51298) 2023-10-06T01:42:33.670 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.668+0000 7fe756d9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fe7580c6dc0 0x7fe7580ae5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48340/0 (socket says 172.21.15.27:48340) 2023-10-06T01:42:33.670 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.668+0000 7fe755d9b700 1 -- 172.21.15.27:0/1866789163 learned_addr learned my addr 172.21.15.27:0/1866789163 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:33.670 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.668+0000 7fe75659c700 1 --2- 172.21.15.27:0/1866789163 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe7580cdc00 0x7fe7580cbe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:33.670 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.668+0000 7fe756d9d700 1 -- 172.21.15.27:0/1866789163 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe7580cdc00 msgr2=0x7fe7580cbe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:33.670 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.668+0000 7fe756d9d700 1 --2- 172.21.15.27:0/1866789163 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe7580cdc00 0x7fe7580cbe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:33.670 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.668+0000 7fe756d9d700 1 -- 172.21.15.27:0/1866789163 >> v1:172.21.15.27:6789/0 conn(0x7fe7580d1030 legacy=0x7fe7580c66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:33.670 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.668+0000 7fe75559a700 1 -- 172.21.15.27:0/1866789163 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3188403801 0 0) 0x7fe7580ae140 con 0x7fe7580d1030 2023-10-06T01:42:33.670 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.668+0000 7fe756d9d700 1 -- 172.21.15.27:0/1866789163 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7580ad810 con 0x7fe7580c6dc0 2023-10-06T01:42:33.670 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.668+0000 7fe756d9d700 1 --2- 172.21.15.27:0/1866789163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7580c6dc0 0x7fe7580ae5d0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fe748009910 tx=0x7fe7480091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b38a90714dfa9aaa server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:33.671 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.668+0000 7fe75559a700 1 -- 172.21.15.27:0/1866789163 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74800b640 con 0x7fe7580c6dc0 2023-10-06T01:42:33.671 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.668+0000 7fe75559a700 1 -- 172.21.15.27:0/1866789163 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe74800b7a0 con 0x7fe7580c6dc0 2023-10-06T01:42:33.671 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.668+0000 7fe75559a700 1 -- 172.21.15.27:0/1866789163 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74800bb60 con 0x7fe7580c6dc0 2023-10-06T01:42:33.671 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.669+0000 7fe75cc40700 1 -- 172.21.15.27:0/1866789163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7580c6dc0 msgr2=0x7fe7580ae5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:33.671 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.669+0000 7fe75cc40700 1 --2- 172.21.15.27:0/1866789163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7580c6dc0 0x7fe7580ae5d0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fe748009910 tx=0x7fe7480091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:33.671 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.669+0000 7fe75cc40700 1 -- 172.21.15.27:0/1866789163 shutdown_connections 2023-10-06T01:42:33.671 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.669+0000 7fe75cc40700 1 --2- 172.21.15.27:0/1866789163 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7580c6dc0 0x7fe7580ae5d0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:33.671 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.669+0000 7fe75cc40700 1 --2- 172.21.15.27:0/1866789163 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe7580cdc00 0x7fe7580cbe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:33.671 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.669+0000 7fe75cc40700 1 -- 172.21.15.27:0/1866789163 >> 172.21.15.27:0/1866789163 conn(0x7fe758099990 msgr2=0x7fe7580b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:33.671 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.669+0000 7fe75cc40700 1 -- 172.21.15.27:0/1866789163 shutdown_connections 2023-10-06T01:42:33.671 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.669+0000 7fe75cc40700 1 -- 172.21.15.27:0/1866789163 wait complete. 2023-10-06T01:42:33.674 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.670+0000 7fe75cc40700 1 Processor -- start 2023-10-06T01:42:33.674 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.670+0000 7fe75cc40700 1 -- start start 2023-10-06T01:42:33.674 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.671+0000 7fe75cc40700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe7580c6dc0 0x7fe758159310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:33.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.671+0000 7fe75cc40700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7580cdc00 0x7fe758067ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:33.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.671+0000 7fe75cc40700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe7580d1030 0x7fe758160070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:33.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.671+0000 7fe75cc40700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe758068730 con 0x7fe7580cdc00 2023-10-06T01:42:33.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.671+0000 7fe75cc40700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe758159850 con 0x7fe7580d1030 2023-10-06T01:42:33.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.671+0000 7fe75cc40700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7580b02d0 con 0x7fe7580c6dc0 2023-10-06T01:42:33.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.671+0000 7fe755d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7580cdc00 0x7fe758067ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:33.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.671+0000 7fe755d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7580cdc00 0x7fe758067ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48354/0 (socket says 172.21.15.27:48354) 2023-10-06T01:42:33.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.671+0000 7fe755d9b700 1 -- 172.21.15.27:0/1156740956 learned_addr learned my addr 172.21.15.27:0/1156740956 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:33.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.671+0000 7fe75659c700 1 --2- 172.21.15.27:0/1156740956 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe7580c6dc0 0x7fe758159310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:33.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.671+0000 7fe756d9d700 1 --2- 172.21.15.27:0/1156740956 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe7580d1030 0x7fe758160070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:33.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.671+0000 7fe755d9b700 1 -- 172.21.15.27:0/1156740956 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe7580c6dc0 msgr2=0x7fe758159310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:33.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.671+0000 7fe755d9b700 1 --2- 172.21.15.27:0/1156740956 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe7580c6dc0 0x7fe758159310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:33.676 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.671+0000 7fe755d9b700 1 -- 172.21.15.27:0/1156740956 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe7580d1030 msgr2=0x7fe758160070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:33.676 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.671+0000 7fe755d9b700 1 --2- 172.21.15.27:0/1156740956 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe7580d1030 0x7fe758160070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:33.676 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.671+0000 7fe755d9b700 1 -- 172.21.15.27:0/1156740956 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe748009420 con 0x7fe7580cdc00 2023-10-06T01:42:33.676 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.671+0000 7fe756d9d700 1 --2- 172.21.15.27:0/1156740956 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe7580d1030 0x7fe758160070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:42:33.676 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.671+0000 7fe755d9b700 1 --2- 172.21.15.27:0/1156740956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7580cdc00 0x7fe758067ee0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fe74000c5b0 tx=0x7fe740001070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:33.676 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.672+0000 7fe74e7fc700 1 -- 172.21.15.27:0/1156740956 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe740005b30 con 0x7fe7580cdc00 2023-10-06T01:42:33.676 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.672+0000 7fe74e7fc700 1 -- 172.21.15.27:0/1156740956 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe740005c90 con 0x7fe7580cdc00 2023-10-06T01:42:33.676 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.672+0000 7fe75cc40700 1 -- 172.21.15.27:0/1156740956 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7580b0570 con 0x7fe7580cdc00 2023-10-06T01:42:33.676 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.672+0000 7fe74e7fc700 1 -- 172.21.15.27:0/1156740956 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe740007300 con 0x7fe7580cdc00 2023-10-06T01:42:33.676 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.672+0000 7fe75cc40700 1 -- 172.21.15.27:0/1156740956 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7580b0b30 con 0x7fe7580cdc00 2023-10-06T01:42:33.676 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.672+0000 7fe75cc40700 1 -- 172.21.15.27:0/1156740956 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe758064050 con 0x7fe7580cdc00 2023-10-06T01:42:33.676 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.673+0000 7fe74e7fc700 1 -- 172.21.15.27:0/1156740956 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fe74001b3b0 con 0x7fe7580cdc00 2023-10-06T01:42:33.677 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.673+0000 7fe74e7fc700 1 --2- 172.21.15.27:0/1156740956 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe738066f30 0x7fe7380693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:33.677 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.673+0000 7fe74e7fc700 1 -- 172.21.15.27:0/1156740956 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5292+0+0 (secure 0 0 0) 0x7fe7400838c0 con 0x7fe7580cdc00 2023-10-06T01:42:33.677 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.674+0000 7fe75659c700 1 --2- 172.21.15.27:0/1156740956 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe738066f30 0x7fe7380693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:33.677 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.674+0000 7fe75659c700 1 --2- 172.21.15.27:0/1156740956 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe738066f30 0x7fe7380693e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe750009fd0 tx=0x7fe75000b040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:33.678 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.678+0000 7fe74e7fc700 1 -- 172.21.15.27:0/1156740956 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe740050cd0 con 0x7fe7580cdc00 2023-10-06T01:42:33.867 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.867+0000 7fe75cc40700 1 -- 172.21.15.27:0/1156740956 --> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe758070790 con 0x7fe738066f30 2023-10-06T01:42:33.870 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.870+0000 7fe74e7fc700 1 -- 172.21.15.27:0/1156740956 <== mgr.4100 v2:172.21.15.124:6832/90278 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+73220 (secure 0 0 0) 0x7fe758070790 con 0x7fe738066f30 2023-10-06T01:42:33.871 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:33.872 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.872+0000 7fe75cc40700 1 -- 172.21.15.27:0/1156740956 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe738066f30 msgr2=0x7fe7380693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:33.873 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.872+0000 7fe75cc40700 1 --2- 172.21.15.27:0/1156740956 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe738066f30 0x7fe7380693e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe750009fd0 tx=0x7fe75000b040 comp rx=0 tx=0).stop 2023-10-06T01:42:33.873 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.872+0000 7fe75cc40700 1 -- 172.21.15.27:0/1156740956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7580cdc00 msgr2=0x7fe758067ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:33.873 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.872+0000 7fe75cc40700 1 --2- 172.21.15.27:0/1156740956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7580cdc00 0x7fe758067ee0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fe74000c5b0 tx=0x7fe740001070 comp rx=0 tx=0).stop 2023-10-06T01:42:33.873 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.872+0000 7fe75cc40700 1 -- 172.21.15.27:0/1156740956 shutdown_connections 2023-10-06T01:42:33.873 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.872+0000 7fe75cc40700 1 --2- 172.21.15.27:0/1156740956 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fe738066f30 0x7fe7380693e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:33.873 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.872+0000 7fe75cc40700 1 --2- 172.21.15.27:0/1156740956 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fe7580c6dc0 0x7fe758159310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:33.873 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.872+0000 7fe75cc40700 1 --2- 172.21.15.27:0/1156740956 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fe7580d1030 0x7fe758160070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:33.873 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.872+0000 7fe75cc40700 1 --2- 172.21.15.27:0/1156740956 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fe7580cdc00 0x7fe758067ee0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:33.873 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.872+0000 7fe75cc40700 1 -- 172.21.15.27:0/1156740956 >> 172.21.15.27:0/1156740956 conn(0x7fe758099990 msgr2=0x7fe75815f060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:33.873 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.872+0000 7fe75cc40700 1 -- 172.21.15.27:0/1156740956 shutdown_connections 2023-10-06T01:42:33.873 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:33.872+0000 7fe75cc40700 1 -- 172.21.15.27:0/1156740956 wait complete. 2023-10-06T01:42:33.874 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-06T01:42:33.884 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2023-10-06T01:42:32.057240+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":143,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2382872,"kb_used_data":4568,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752591848,"statfs":{"total":773094113280,"available":770654052352,"internally_reserved":0,"allocated":4677632,"data_stored":2655194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-58022,"num_objects":-33,"num_object_clones":0,"num_object_copies":-66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-33,"num_whiteouts":0,"num_read":-22,"num_read_kb":-11,"num_write":-185,"num_write_kb":-131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-19,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.944022"},"pg_stats":[{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-06T01:42:29.422519+0000","last_change":"2023-10-06T01:41:07.454295+0000","last_active":"2023-10-06T01:42:29.422519+0000","last_peered":"2023-10-06T01:42:29.422519+0000","last_clean":"2023-10-06T01:42:29.422519+0000","last_became_active":"2023-10-06T01:41:07.454034+0000","last_became_peered":"2023-10-06T01:41:07.454034+0000","last_unstale":"2023-10-06T01:42:29.422519+0000","last_undegraded":"2023-10-06T01:42:29.422519+0000","last_fullsized":"2023-10-06T01:42:29.422519+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:32:58.914183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":30,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-06T01:42:28.418774+0000","last_change":"2023-10-06T01:41:07.261955+0000","last_active":"2023-10-06T01:42:28.418774+0000","last_peered":"2023-10-06T01:42:28.418774+0000","last_clean":"2023-10-06T01:42:28.418774+0000","last_became_active":"2023-10-06T01:41:07.261692+0000","last_became_peered":"2023-10-06T01:41:07.261692+0000","last_unstale":"2023-10-06T01:42:28.418774+0000","last_undegraded":"2023-10-06T01:42:28.418774+0000","last_fullsized":"2023-10-06T01:42:28.418774+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":27,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-06T01:42:28.773478+0000","last_change":"2023-10-06T01:41:07.258362+0000","last_active":"2023-10-06T01:42:28.773478+0000","last_peered":"2023-10-06T01:42:28.773478+0000","last_clean":"2023-10-06T01:42:28.773478+0000","last_became_active":"2023-10-06T01:41:07.258055+0000","last_became_peered":"2023-10-06T01:41:07.258055+0000","last_unstale":"2023-10-06T01:42:28.773478+0000","last_undegraded":"2023-10-06T01:42:28.773478+0000","last_fullsized":"2023-10-06T01:42:28.773478+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"16'117","reported_seq":178,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-06T01:42:29.513225+0000","last_change":"2023-10-06T01:41:07.261164+0000","last_active":"2023-10-06T01:42:29.513225+0000","last_peered":"2023-10-06T01:42:29.513225+0000","last_clean":"2023-10-06T01:42:29.513225+0000","last_became_active":"2023-10-06T01:41:07.260896+0000","last_became_peered":"2023-10-06T01:41:07.260896+0000","last_unstale":"2023-10-06T01:42:29.513225+0000","last_undegraded":"2023-10-06T01:42:29.513225+0000","last_fullsized":"2023-10-06T01:42:29.513225+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:05:22.934401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":28,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-06T01:42:29.515247+0000","last_change":"2023-10-06T01:41:07.456511+0000","last_active":"2023-10-06T01:42:29.515247+0000","last_peered":"2023-10-06T01:42:29.515247+0000","last_clean":"2023-10-06T01:42:29.515247+0000","last_became_active":"2023-10-06T01:41:07.456024+0000","last_became_peered":"2023-10-06T01:41:07.456024+0000","last_unstale":"2023-10-06T01:42:29.515247+0000","last_undegraded":"2023-10-06T01:42:29.515247+0000","last_fullsized":"2023-10-06T01:42:29.515247+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:51:02.949231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":28,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-06T01:42:29.516306+0000","last_change":"2023-10-06T01:41:07.456273+0000","last_active":"2023-10-06T01:42:29.516306+0000","last_peered":"2023-10-06T01:42:29.516306+0000","last_clean":"2023-10-06T01:42:29.516306+0000","last_became_active":"2023-10-06T01:41:07.456000+0000","last_became_peered":"2023-10-06T01:41:07.456000+0000","last_unstale":"2023-10-06T01:42:29.516306+0000","last_undegraded":"2023-10-06T01:42:29.516306+0000","last_fullsized":"2023-10-06T01:42:29.516306+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:01:42.633388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":29,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-06T01:42:29.518007+0000","last_change":"2023-10-06T01:41:07.456688+0000","last_active":"2023-10-06T01:42:29.518007+0000","last_peered":"2023-10-06T01:42:29.518007+0000","last_clean":"2023-10-06T01:42:29.518007+0000","last_became_active":"2023-10-06T01:41:07.456304+0000","last_became_peered":"2023-10-06T01:41:07.456304+0000","last_unstale":"2023-10-06T01:42:29.518007+0000","last_undegraded":"2023-10-06T01:42:29.518007+0000","last_fullsized":"2023-10-06T01:42:29.518007+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":30,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-06T01:42:29.516302+0000","last_change":"2023-10-06T01:41:07.456388+0000","last_active":"2023-10-06T01:42:29.516302+0000","last_peered":"2023-10-06T01:42:29.516302+0000","last_clean":"2023-10-06T01:42:29.516302+0000","last_became_active":"2023-10-06T01:41:07.456045+0000","last_became_peered":"2023-10-06T01:41:07.456045+0000","last_unstale":"2023-10-06T01:42:29.516302+0000","last_undegraded":"2023-10-06T01:42:29.516302+0000","last_fullsized":"2023-10-06T01:42:29.516302+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":30,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-06T01:42:29.517621+0000","last_change":"2023-10-06T01:41:07.258162+0000","last_active":"2023-10-06T01:42:29.517621+0000","last_peered":"2023-10-06T01:42:29.517621+0000","last_clean":"2023-10-06T01:42:29.517621+0000","last_became_active":"2023-10-06T01:41:07.257795+0000","last_became_peered":"2023-10-06T01:41:07.257795+0000","last_unstale":"2023-10-06T01:42:29.517621+0000","last_undegraded":"2023-10-06T01:42:29.517621+0000","last_fullsized":"2023-10-06T01:42:29.517621+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:05:22.934401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298304,"kb_used_data":1016,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073536,"statfs":{"total":96636764160,"available":96331300864,"internally_reserved":0,"allocated":1040384,"data_stored":778347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 01:42:08 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.51000000000000001}]},{"osd":1,"last update":"Fri Oct 6 01:42:08 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.49299999999999999}]},{"osd":2,"last update":"Fri Oct 6 01:42:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.622}]},{"osd":3,"last update":"Fri Oct 6 01:42:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.55000000000000004}]},{"osd":4,"last update":"Fri Oct 6 01:42:08 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.65000000000000002}]},{"osd":5,"last update":"Fri Oct 6 01:42:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.60599999999999998}]},{"osd":6,"last update":"Fri Oct 6 01:42:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.58399999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297720,"kb_used_data":432,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074120,"statfs":{"total":96636764160,"available":96331898880,"internally_reserved":0,"allocated":442368,"data_stored":186626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 01:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.63900000000000001}]},{"osd":1,"last update":"Fri Oct 6 01:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.628}]},{"osd":2,"last update":"Fri Oct 6 01:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.78500000000000003}]},{"osd":3,"last update":"Fri Oct 6 01:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.66200000000000003}]},{"osd":4,"last update":"Fri Oct 6 01:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.51700000000000002}]},{"osd":5,"last update":"Fri Oct 6 01:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.501}]},{"osd":7,"last update":"Fri Oct 6 01:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.442},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.55200000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298288,"kb_used_data":1000,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073552,"statfs":{"total":96636764160,"available":96331317248,"internally_reserved":0,"allocated":1024000,"data_stored":772460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.255},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.27400000000000002}]},{"osd":2,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.318}]},{"osd":3,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.34100000000000003}]},{"osd":4,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.56899999999999995}]},{"osd":5,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.496}]},{"osd":6,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.59399999999999997}]},{"osd":7,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.55400000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297688,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074152,"statfs":{"total":96636764160,"available":96331931648,"internally_reserved":0,"allocated":409600,"data_stored":174438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.495,"5min":0.495,"15min":0.495},"last":0.318},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"last":0.215}]},{"osd":2,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"last":0.29699999999999999}]},{"osd":3,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.439,"5min":0.439,"15min":0.439},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.34799999999999998}]},{"osd":4,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.56899999999999995}]},{"osd":5,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.55500000000000005}]},{"osd":6,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.505}]},{"osd":7,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.49099999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574870,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297724,"kb_used_data":436,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074116,"statfs":{"total":96636764160,"available":96331894784,"internally_reserved":0,"allocated":446464,"data_stored":187979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.497,"5min":0.497,"15min":0.497},"last":0.26700000000000002}]},{"osd":1,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.35499999999999998}]},{"osd":3,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.216},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.313}]},{"osd":4,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.56999999999999995}]},{"osd":5,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.63200000000000001}]},{"osd":6,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.51400000000000001}]},{"osd":7,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.502}]}]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297704,"kb_used_data":416,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074136,"statfs":{"total":96636764160,"available":96331915264,"internally_reserved":0,"allocated":425984,"data_stored":180739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.31,"15min":0.31},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"last":0.39700000000000002}]},{"osd":1,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.42999999999999999}]},{"osd":2,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"last":0.48799999999999999}]},{"osd":4,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.64100000000000001}]},{"osd":5,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.68000000000000005}]},{"osd":6,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.72199999999999998}]},{"osd":7,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.69099999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297724,"kb_used_data":436,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074116,"statfs":{"total":96636764160,"available":96331894784,"internally_reserved":0,"allocated":446464,"data_stored":187979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.61399999999999999}]},{"osd":1,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.58799999999999997}]},{"osd":2,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.52700000000000002}]},{"osd":3,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.75}]},{"osd":5,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.46600000000000003}]},{"osd":6,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.77900000000000003}]},{"osd":7,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.55000000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297720,"kb_used_data":432,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074120,"statfs":{"total":96636764160,"available":96331898880,"internally_reserved":0,"allocated":442368,"data_stored":186626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.335,"5min":1.335,"15min":1.335},"last":0.437}]},{"osd":1,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":2.4849999999999999,"5min":2.4849999999999999,"15min":2.4849999999999999},"last":0.55900000000000005}]},{"osd":2,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.8280000000000001,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":0.45900000000000002}]},{"osd":3,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.9830000000000001,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":0.5},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.36799999999999999}]},{"osd":4,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.45100000000000001}]},{"osd":6,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.34599999999999997}]},{"osd":7,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.47099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-06T01:42:33.887 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-06T01:42:34.005 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.005+0000 7f1967e27700 1 Processor -- start 2023-10-06T01:42:34.006 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.005+0000 7f1967e27700 1 -- start start 2023-10-06T01:42:34.007 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.006+0000 7f1967e27700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f19600cbe50 0x7f19600c66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:34.007 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.006+0000 7f1967e27700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19600ae140 con 0x7f19600cdc00 2023-10-06T01:42:34.007 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.006+0000 7f1967e27700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19600c7020 con 0x7f19600c6bf0 2023-10-06T01:42:34.007 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.006+0000 7f1967e27700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19600cc2a0 con 0x7f19600cbe50 2023-10-06T01:42:34.007 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.006+0000 7f19653c2700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f19600cbe50 0x7f19600c66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:34.008 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.006+0000 7f1965bc3700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f19600cdc00 0x7f19600d1030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51308/0 (socket says 172.21.15.27:51308) 2023-10-06T01:42:34.008 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.006+0000 7f1965bc3700 1 -- 172.21.15.27:0/888049156 learned_addr learned my addr 172.21.15.27:0/888049156 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:34.008 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.007+0000 7f19653c2700 1 -- 172.21.15.27:0/888049156 >> v1:172.21.15.124:6789/0 conn(0x7f19600c6bf0 legacy=0x7f19600ae5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:34.008 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.007+0000 7f19653c2700 1 -- 172.21.15.27:0/888049156 >> v1:172.21.15.27:6789/0 conn(0x7f19600cdc00 legacy=0x7f19600d1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:34.008 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.007+0000 7f19653c2700 1 -- 172.21.15.27:0/888049156 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19600ad530 con 0x7f19600cbe50 2023-10-06T01:42:34.008 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.007+0000 7f1964bc1700 1 -- 172.21.15.27:0/888049156 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3584574685 0 0) 0x7f19600ae140 con 0x7f19600cdc00 2023-10-06T01:42:34.008 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.007+0000 7f19653c2700 1 --2- 172.21.15.27:0/888049156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19600cbe50 0x7f19600c66b0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f19540098e0 tx=0x7f19540091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe0c032415186cc7 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:34.008 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.007+0000 7f1964bc1700 1 -- 172.21.15.27:0/888049156 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1954009e80 con 0x7f19600cbe50 2023-10-06T01:42:34.008 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.007+0000 7f1964bc1700 1 -- 172.21.15.27:0/888049156 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f195400b860 con 0x7f19600cbe50 2023-10-06T01:42:34.008 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.007+0000 7f1964bc1700 1 -- 172.21.15.27:0/888049156 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195400bbe0 con 0x7f19600cbe50 2023-10-06T01:42:34.011 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.007+0000 7f1967e27700 1 -- 172.21.15.27:0/888049156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19600cbe50 msgr2=0x7f19600c66b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:34.011 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.007+0000 7f1967e27700 1 --2- 172.21.15.27:0/888049156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19600cbe50 0x7f19600c66b0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f19540098e0 tx=0x7f19540091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:34.011 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.007+0000 7f1967e27700 1 -- 172.21.15.27:0/888049156 shutdown_connections 2023-10-06T01:42:34.011 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.007+0000 7f1967e27700 1 --2- 172.21.15.27:0/888049156 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19600cbe50 0x7f19600c66b0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:34.011 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.007+0000 7f1967e27700 1 -- 172.21.15.27:0/888049156 >> 172.21.15.27:0/888049156 conn(0x7f1960099990 msgr2=0x7f19600b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:34.011 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.008+0000 7f1967e27700 1 -- 172.21.15.27:0/888049156 shutdown_connections 2023-10-06T01:42:34.011 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.008+0000 7f1967e27700 1 -- 172.21.15.27:0/888049156 wait complete. 2023-10-06T01:42:34.011 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.009+0000 7f1967e27700 1 Processor -- start 2023-10-06T01:42:34.011 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.009+0000 7f1967e27700 1 -- start start 2023-10-06T01:42:34.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.009+0000 7f1967e27700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f19600c6bf0 0x7f1960148f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:34.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.009+0000 7f1967e27700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19600cbe50 0x7f1960145ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:34.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.009+0000 7f1967e27700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f19600cdc00 0x7f1960147050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:34.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.009+0000 7f1967e27700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1960148a70 con 0x7f19600cbe50 2023-10-06T01:42:34.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.009+0000 7f1967e27700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1960147e60 con 0x7f19600c6bf0 2023-10-06T01:42:34.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.009+0000 7f1967e27700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1960148140 con 0x7f19600cdc00 2023-10-06T01:42:34.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.009+0000 7f19653c2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19600cbe50 0x7f1960145ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:34.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.010+0000 7f19653c2700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19600cbe50 0x7f1960145ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48382/0 (socket says 172.21.15.27:48382) 2023-10-06T01:42:34.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.010+0000 7f19653c2700 1 -- 172.21.15.27:0/1908687777 learned_addr learned my addr 172.21.15.27:0/1908687777 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:34.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.010+0000 7f1965bc3700 1 --2- 172.21.15.27:0/1908687777 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f19600c6bf0 0x7f1960148f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:34.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.010+0000 7f19663c4700 1 --2- 172.21.15.27:0/1908687777 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f19600cdc00 0x7f1960147050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:34.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.010+0000 7f19653c2700 1 -- 172.21.15.27:0/1908687777 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f19600cdc00 msgr2=0x7f1960147050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:34.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.010+0000 7f19653c2700 1 --2- 172.21.15.27:0/1908687777 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f19600cdc00 0x7f1960147050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:34.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.010+0000 7f19653c2700 1 -- 172.21.15.27:0/1908687777 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f19600c6bf0 msgr2=0x7f1960148f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:34.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.010+0000 7f19653c2700 1 --2- 172.21.15.27:0/1908687777 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f19600c6bf0 0x7f1960148f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:34.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.010+0000 7f19653c2700 1 -- 172.21.15.27:0/1908687777 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1954009420 con 0x7f19600cbe50 2023-10-06T01:42:34.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.010+0000 7f19663c4700 1 --2- 172.21.15.27:0/1908687777 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f19600cdc00 0x7f1960147050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:42:34.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.010+0000 7f19653c2700 1 --2- 172.21.15.27:0/1908687777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19600cbe50 0x7f1960145ad0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f19600889c0 tx=0x7f1954017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:34.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.010+0000 7f195dffb700 1 -- 172.21.15.27:0/1908687777 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195401f070 con 0x7f19600cbe50 2023-10-06T01:42:34.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.010+0000 7f195dffb700 1 -- 172.21.15.27:0/1908687777 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1954000c90 con 0x7f19600cbe50 2023-10-06T01:42:34.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.010+0000 7f1967e27700 1 -- 172.21.15.27:0/1908687777 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1960148420 con 0x7f19600cbe50 2023-10-06T01:42:34.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.010+0000 7f195dffb700 1 -- 172.21.15.27:0/1908687777 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1954025420 con 0x7f19600cbe50 2023-10-06T01:42:34.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.010+0000 7f1967e27700 1 -- 172.21.15.27:0/1908687777 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1960146010 con 0x7f19600cbe50 2023-10-06T01:42:34.014 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.011+0000 7f1967e27700 1 -- 172.21.15.27:0/1908687777 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1960148be0 con 0x7f19600cbe50 2023-10-06T01:42:34.014 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.012+0000 7f195dffb700 1 -- 172.21.15.27:0/1908687777 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f1954023070 con 0x7f19600cbe50 2023-10-06T01:42:34.014 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.012+0000 7f195dffb700 1 --2- 172.21.15.27:0/1908687777 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f194c066f60 0x7f194c069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:34.014 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.012+0000 7f195dffb700 1 -- 172.21.15.27:0/1908687777 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5292+0+0 (secure 0 0 0) 0x7f19540856b0 con 0x7f19600cbe50 2023-10-06T01:42:34.015 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.012+0000 7f1965bc3700 1 --2- 172.21.15.27:0/1908687777 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f194c066f60 0x7f194c069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:34.015 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.013+0000 7f1965bc3700 1 --2- 172.21.15.27:0/1908687777 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f194c066f60 0x7f194c069410 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f1958000c00 tx=0x7f1958009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:34.016 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.016+0000 7f195dffb700 1 -- 172.21.15.27:0/1908687777 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1954052ac0 con 0x7f19600cbe50 2023-10-06T01:42:34.205 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.205+0000 7f1967e27700 1 -- 172.21.15.27:0/1908687777 --> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1960064050 con 0x7f194c066f60 2023-10-06T01:42:34.208 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.208+0000 7f195dffb700 1 -- 172.21.15.27:0/1908687777 <== mgr.4100 v2:172.21.15.124:6832/90278 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+73220 (secure 0 0 0) 0x7f1960064050 con 0x7f194c066f60 2023-10-06T01:42:34.209 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:34.210 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.210+0000 7f1967e27700 1 -- 172.21.15.27:0/1908687777 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f194c066f60 msgr2=0x7f194c069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.210+0000 7f1967e27700 1 --2- 172.21.15.27:0/1908687777 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f194c066f60 0x7f194c069410 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f1958000c00 tx=0x7f1958009040 comp rx=0 tx=0).stop 2023-10-06T01:42:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.210+0000 7f1967e27700 1 -- 172.21.15.27:0/1908687777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19600cbe50 msgr2=0x7f1960145ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.210+0000 7f1967e27700 1 --2- 172.21.15.27:0/1908687777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19600cbe50 0x7f1960145ad0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f19600889c0 tx=0x7f1954017000 comp rx=0 tx=0).stop 2023-10-06T01:42:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.210+0000 7f1967e27700 1 -- 172.21.15.27:0/1908687777 shutdown_connections 2023-10-06T01:42:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.210+0000 7f1967e27700 1 --2- 172.21.15.27:0/1908687777 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f194c066f60 0x7f194c069410 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.210+0000 7f1967e27700 1 --2- 172.21.15.27:0/1908687777 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f19600cdc00 0x7f1960147050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.210+0000 7f1967e27700 1 --2- 172.21.15.27:0/1908687777 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f19600c6bf0 0x7f1960148f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.210+0000 7f1967e27700 1 --2- 172.21.15.27:0/1908687777 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f19600cbe50 0x7f1960145ad0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.210+0000 7f1967e27700 1 -- 172.21.15.27:0/1908687777 >> 172.21.15.27:0/1908687777 conn(0x7f1960099990 msgr2=0x7f1960158f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.210+0000 7f1967e27700 1 -- 172.21.15.27:0/1908687777 shutdown_connections 2023-10-06T01:42:34.211 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.210+0000 7f1967e27700 1 -- 172.21.15.27:0/1908687777 wait complete. 2023-10-06T01:42:34.212 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-06T01:42:34.222 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":75,"stamp":"2023-10-06T01:42:34.057618+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":143,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2382872,"kb_used_data":4568,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752591848,"statfs":{"total":773094113280,"available":770654052352,"internally_reserved":0,"allocated":4677632,"data_stored":2655194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-58022,"num_objects":-33,"num_object_clones":0,"num_object_copies":-66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-33,"num_whiteouts":0,"num_read":-22,"num_read_kb":-11,"num_write":-187,"num_write_kb":-131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-19,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001454"},"pg_stats":[{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-06T01:42:29.422519+0000","last_change":"2023-10-06T01:41:07.454295+0000","last_active":"2023-10-06T01:42:29.422519+0000","last_peered":"2023-10-06T01:42:29.422519+0000","last_clean":"2023-10-06T01:42:29.422519+0000","last_became_active":"2023-10-06T01:41:07.454034+0000","last_became_peered":"2023-10-06T01:41:07.454034+0000","last_unstale":"2023-10-06T01:42:29.422519+0000","last_undegraded":"2023-10-06T01:42:29.422519+0000","last_fullsized":"2023-10-06T01:42:29.422519+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:32:58.914183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":30,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-06T01:42:28.418774+0000","last_change":"2023-10-06T01:41:07.261955+0000","last_active":"2023-10-06T01:42:28.418774+0000","last_peered":"2023-10-06T01:42:28.418774+0000","last_clean":"2023-10-06T01:42:28.418774+0000","last_became_active":"2023-10-06T01:41:07.261692+0000","last_became_peered":"2023-10-06T01:41:07.261692+0000","last_unstale":"2023-10-06T01:42:28.418774+0000","last_undegraded":"2023-10-06T01:42:28.418774+0000","last_fullsized":"2023-10-06T01:42:28.418774+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":27,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-06T01:42:28.773478+0000","last_change":"2023-10-06T01:41:07.258362+0000","last_active":"2023-10-06T01:42:28.773478+0000","last_peered":"2023-10-06T01:42:28.773478+0000","last_clean":"2023-10-06T01:42:28.773478+0000","last_became_active":"2023-10-06T01:41:07.258055+0000","last_became_peered":"2023-10-06T01:41:07.258055+0000","last_unstale":"2023-10-06T01:42:28.773478+0000","last_undegraded":"2023-10-06T01:42:28.773478+0000","last_fullsized":"2023-10-06T01:42:28.773478+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"16'117","reported_seq":178,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-06T01:42:29.513225+0000","last_change":"2023-10-06T01:41:07.261164+0000","last_active":"2023-10-06T01:42:29.513225+0000","last_peered":"2023-10-06T01:42:29.513225+0000","last_clean":"2023-10-06T01:42:29.513225+0000","last_became_active":"2023-10-06T01:41:07.260896+0000","last_became_peered":"2023-10-06T01:41:07.260896+0000","last_unstale":"2023-10-06T01:42:29.513225+0000","last_undegraded":"2023-10-06T01:42:29.513225+0000","last_fullsized":"2023-10-06T01:42:29.513225+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:05:22.934401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":28,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-06T01:42:29.515247+0000","last_change":"2023-10-06T01:41:07.456511+0000","last_active":"2023-10-06T01:42:29.515247+0000","last_peered":"2023-10-06T01:42:29.515247+0000","last_clean":"2023-10-06T01:42:29.515247+0000","last_became_active":"2023-10-06T01:41:07.456024+0000","last_became_peered":"2023-10-06T01:41:07.456024+0000","last_unstale":"2023-10-06T01:42:29.515247+0000","last_undegraded":"2023-10-06T01:42:29.515247+0000","last_fullsized":"2023-10-06T01:42:29.515247+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:51:02.949231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":28,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-06T01:42:29.516306+0000","last_change":"2023-10-06T01:41:07.456273+0000","last_active":"2023-10-06T01:42:29.516306+0000","last_peered":"2023-10-06T01:42:29.516306+0000","last_clean":"2023-10-06T01:42:29.516306+0000","last_became_active":"2023-10-06T01:41:07.456000+0000","last_became_peered":"2023-10-06T01:41:07.456000+0000","last_unstale":"2023-10-06T01:42:29.516306+0000","last_undegraded":"2023-10-06T01:42:29.516306+0000","last_fullsized":"2023-10-06T01:42:29.516306+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:01:42.633388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":29,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-06T01:42:29.518007+0000","last_change":"2023-10-06T01:41:07.456688+0000","last_active":"2023-10-06T01:42:29.518007+0000","last_peered":"2023-10-06T01:42:29.518007+0000","last_clean":"2023-10-06T01:42:29.518007+0000","last_became_active":"2023-10-06T01:41:07.456304+0000","last_became_peered":"2023-10-06T01:41:07.456304+0000","last_unstale":"2023-10-06T01:42:29.518007+0000","last_undegraded":"2023-10-06T01:42:29.518007+0000","last_fullsized":"2023-10-06T01:42:29.518007+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":30,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-06T01:42:29.516302+0000","last_change":"2023-10-06T01:41:07.456388+0000","last_active":"2023-10-06T01:42:29.516302+0000","last_peered":"2023-10-06T01:42:29.516302+0000","last_clean":"2023-10-06T01:42:29.516302+0000","last_became_active":"2023-10-06T01:41:07.456045+0000","last_became_peered":"2023-10-06T01:41:07.456045+0000","last_unstale":"2023-10-06T01:42:29.516302+0000","last_undegraded":"2023-10-06T01:42:29.516302+0000","last_fullsized":"2023-10-06T01:42:29.516302+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":30,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-06T01:42:29.517621+0000","last_change":"2023-10-06T01:41:07.258162+0000","last_active":"2023-10-06T01:42:29.517621+0000","last_peered":"2023-10-06T01:42:29.517621+0000","last_clean":"2023-10-06T01:42:29.517621+0000","last_became_active":"2023-10-06T01:41:07.257795+0000","last_became_peered":"2023-10-06T01:41:07.257795+0000","last_unstale":"2023-10-06T01:42:29.517621+0000","last_undegraded":"2023-10-06T01:42:29.517621+0000","last_fullsized":"2023-10-06T01:42:29.517621+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:05:22.934401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298304,"kb_used_data":1016,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073536,"statfs":{"total":96636764160,"available":96331300864,"internally_reserved":0,"allocated":1040384,"data_stored":778347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 01:42:08 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.51000000000000001}]},{"osd":1,"last update":"Fri Oct 6 01:42:08 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.49299999999999999}]},{"osd":2,"last update":"Fri Oct 6 01:42:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.622}]},{"osd":3,"last update":"Fri Oct 6 01:42:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.55000000000000004}]},{"osd":4,"last update":"Fri Oct 6 01:42:08 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.65000000000000002}]},{"osd":5,"last update":"Fri Oct 6 01:42:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.60599999999999998}]},{"osd":6,"last update":"Fri Oct 6 01:42:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.58399999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297720,"kb_used_data":432,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074120,"statfs":{"total":96636764160,"available":96331898880,"internally_reserved":0,"allocated":442368,"data_stored":186626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 01:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.63900000000000001}]},{"osd":1,"last update":"Fri Oct 6 01:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.628}]},{"osd":2,"last update":"Fri Oct 6 01:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.78500000000000003}]},{"osd":3,"last update":"Fri Oct 6 01:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.66200000000000003}]},{"osd":4,"last update":"Fri Oct 6 01:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.51700000000000002}]},{"osd":5,"last update":"Fri Oct 6 01:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.501}]},{"osd":7,"last update":"Fri Oct 6 01:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.442},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.55200000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298288,"kb_used_data":1000,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073552,"statfs":{"total":96636764160,"available":96331317248,"internally_reserved":0,"allocated":1024000,"data_stored":772460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.255},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.27400000000000002}]},{"osd":2,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.318}]},{"osd":3,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.34100000000000003}]},{"osd":4,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.56899999999999995}]},{"osd":5,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.496}]},{"osd":6,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.59399999999999997}]},{"osd":7,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.55400000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297688,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074152,"statfs":{"total":96636764160,"available":96331931648,"internally_reserved":0,"allocated":409600,"data_stored":174438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.495,"5min":0.495,"15min":0.495},"last":0.318},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"last":0.215}]},{"osd":2,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"last":0.29699999999999999}]},{"osd":3,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.439,"5min":0.439,"15min":0.439},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.34799999999999998}]},{"osd":4,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.56899999999999995}]},{"osd":5,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.55500000000000005}]},{"osd":6,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.505}]},{"osd":7,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.49099999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574870,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297724,"kb_used_data":436,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074116,"statfs":{"total":96636764160,"available":96331894784,"internally_reserved":0,"allocated":446464,"data_stored":187979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.497,"5min":0.497,"15min":0.497},"last":0.26700000000000002}]},{"osd":1,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.35499999999999998}]},{"osd":3,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.216},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.313}]},{"osd":4,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.56999999999999995}]},{"osd":5,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.63200000000000001}]},{"osd":6,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.51400000000000001}]},{"osd":7,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.502}]}]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297704,"kb_used_data":416,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074136,"statfs":{"total":96636764160,"available":96331915264,"internally_reserved":0,"allocated":425984,"data_stored":180739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.31,"15min":0.31},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"last":0.39700000000000002}]},{"osd":1,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.42999999999999999}]},{"osd":2,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"last":0.48799999999999999}]},{"osd":4,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.64100000000000001}]},{"osd":5,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.68000000000000005}]},{"osd":6,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.72199999999999998}]},{"osd":7,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.69099999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297724,"kb_used_data":436,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074116,"statfs":{"total":96636764160,"available":96331894784,"internally_reserved":0,"allocated":446464,"data_stored":187979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.61399999999999999}]},{"osd":1,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.58799999999999997}]},{"osd":2,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.52700000000000002}]},{"osd":3,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.75}]},{"osd":5,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.46600000000000003}]},{"osd":6,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.77900000000000003}]},{"osd":7,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.55000000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297720,"kb_used_data":432,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074120,"statfs":{"total":96636764160,"available":96331898880,"internally_reserved":0,"allocated":442368,"data_stored":186626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.335,"5min":1.335,"15min":1.335},"last":0.437}]},{"osd":1,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":2.4849999999999999,"5min":2.4849999999999999,"15min":2.4849999999999999},"last":0.55900000000000005}]},{"osd":2,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.8280000000000001,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":0.45900000000000002}]},{"osd":3,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.9830000000000001,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":0.5},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.36799999999999999}]},{"osd":4,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.45100000000000001}]},{"osd":6,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.34599999999999997}]},{"osd":7,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.47099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-06T01:42:34.224 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-06T01:42:34.224 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-06T01:42:34.343 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.343+0000 7fb42b579700 1 Processor -- start 2023-10-06T01:42:34.343 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.343+0000 7fb42b579700 1 -- start start 2023-10-06T01:42:34.347 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.344+0000 7fb42b579700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fb4240cc170 0x7fb42415c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:34.347 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.344+0000 7fb42b579700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4240af040 con 0x7fb4240cbda0 2023-10-06T01:42:34.347 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.344+0000 7fb42b579700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4240afd70 con 0x7fb42415cc60 2023-10-06T01:42:34.347 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.344+0000 7fb42b579700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb424068080 con 0x7fb4240cc170 2023-10-06T01:42:34.347 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.344+0000 7fb42a577700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb4240cbda0 0x7fb424067630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51314/0 (socket says 172.21.15.27:51314) 2023-10-06T01:42:34.347 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.344+0000 7fb42a577700 1 -- 172.21.15.27:0/2632019422 learned_addr learned my addr 172.21.15.27:0/2632019422 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:34.348 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.344+0000 7fb429d76700 1 --2- 172.21.15.27:0/2632019422 >> v2:172.21.15.124:3300/0 conn(0x7fb4240cc170 0x7fb42415c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:34.348 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.344+0000 7fb429575700 1 -- 172.21.15.27:0/2632019422 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4169431094 0 0) 0x7fb4240af040 con 0x7fb4240cbda0 2023-10-06T01:42:34.348 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.344+0000 7fb429575700 1 -- 172.21.15.27:0/2632019422 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb40c003570 con 0x7fb4240cbda0 2023-10-06T01:42:34.348 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.344+0000 7fb429575700 1 -- 172.21.15.27:0/2632019422 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4274608519 0 0) 0x7fb40c003570 con 0x7fb4240cbda0 2023-10-06T01:42:34.348 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.344+0000 7fb429575700 1 -- 172.21.15.27:0/2632019422 >> v2:172.21.15.124:3300/0 conn(0x7fb4240cc170 msgr2=0x7fb42415c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:34.348 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.344+0000 7fb429575700 1 --2- 172.21.15.27:0/2632019422 >> v2:172.21.15.124:3300/0 conn(0x7fb4240cc170 0x7fb42415c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:34.348 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.344+0000 7fb429575700 1 -- 172.21.15.27:0/2632019422 >> v1:172.21.15.124:6789/0 conn(0x7fb42415cc60 legacy=0x7fb42415d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:34.348 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.344+0000 7fb429575700 1 -- 172.21.15.27:0/2632019422 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4240af290 con 0x7fb4240cbda0 2023-10-06T01:42:34.348 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.344+0000 7fb429575700 1 -- 172.21.15.27:0/2632019422 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2619078944 0 0) 0x7fb4140017f0 con 0x7fb4240cbda0 2023-10-06T01:42:34.348 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.345+0000 7fb429575700 1 -- 172.21.15.27:0/2632019422 >> v1:172.21.15.27:6789/0 conn(0x7fb4240cbda0 legacy=0x7fb424067630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:34.348 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.345+0000 7fb429575700 1 --2- 172.21.15.27:0/2632019422 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb40c003a60 0x7fb40c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:34.348 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.345+0000 7fb429575700 1 --2- 172.21.15.27:0/2632019422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb40c004810 0x7fb40c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:34.349 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.345+0000 7fb429575700 1 --2- 172.21.15.27:0/2632019422 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb40c0051c0 0x7fb40c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:34.349 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.345+0000 7fb429575700 1 -- 172.21.15.27:0/2632019422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb424068080 con 0x7fb40c004810 2023-10-06T01:42:34.349 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.345+0000 7fb429575700 1 -- 172.21.15.27:0/2632019422 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40c00c0d0 con 0x7fb40c003a60 2023-10-06T01:42:34.349 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.345+0000 7fb429575700 1 -- 172.21.15.27:0/2632019422 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb40c00c290 con 0x7fb40c0051c0 2023-10-06T01:42:34.349 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.345+0000 7fb429d76700 1 --2- 172.21.15.27:0/2632019422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb40c004810 0x7fb40c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:34.349 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.345+0000 7fb42a577700 1 --2- 172.21.15.27:0/2632019422 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb40c003a60 0x7fb40c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:34.349 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.345+0000 7fb42ad78700 1 --2- 172.21.15.27:0/2632019422 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb40c0051c0 0x7fb40c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:34.349 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.345+0000 7fb429d76700 1 -- 172.21.15.27:0/2632019422 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb40c0051c0 msgr2=0x7fb40c00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:34.349 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.345+0000 7fb429d76700 1 --2- 172.21.15.27:0/2632019422 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb40c0051c0 0x7fb40c00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:34.349 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.345+0000 7fb429d76700 1 -- 172.21.15.27:0/2632019422 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb40c003a60 msgr2=0x7fb40c003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:34.349 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.345+0000 7fb429d76700 1 --2- 172.21.15.27:0/2632019422 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb40c003a60 0x7fb40c003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:34.350 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.345+0000 7fb429d76700 1 -- 172.21.15.27:0/2632019422 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb40c003570 con 0x7fb40c004810 2023-10-06T01:42:34.350 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.345+0000 7fb42ad78700 1 --2- 172.21.15.27:0/2632019422 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb40c0051c0 0x7fb40c00ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:42:34.350 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.345+0000 7fb42ad78700 1 -- 172.21.15.27:0/2632019422 reap_dead start 2023-10-06T01:42:34.350 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.345+0000 7fb429d76700 1 --2- 172.21.15.27:0/2632019422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb40c004810 0x7fb40c004c80 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fb418009c00 tx=0x7fb4180091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23773c5ac45a6956 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:34.350 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.345+0000 7fb429575700 1 -- 172.21.15.27:0/2632019422 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41800e710 con 0x7fb40c004810 2023-10-06T01:42:34.350 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.345+0000 7fb429575700 1 -- 172.21.15.27:0/2632019422 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb41800e870 con 0x7fb40c004810 2023-10-06T01:42:34.350 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.345+0000 7fb429575700 1 -- 172.21.15.27:0/2632019422 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41800ec30 con 0x7fb40c004810 2023-10-06T01:42:34.350 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.346+0000 7fb42b579700 1 -- 172.21.15.27:0/2632019422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb40c004810 msgr2=0x7fb40c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:34.350 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.346+0000 7fb42b579700 1 --2- 172.21.15.27:0/2632019422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb40c004810 0x7fb40c004c80 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fb418009c00 tx=0x7fb4180091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:34.350 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.346+0000 7fb42b579700 1 -- 172.21.15.27:0/2632019422 shutdown_connections 2023-10-06T01:42:34.350 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.346+0000 7fb42b579700 1 --2- 172.21.15.27:0/2632019422 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb40c004810 0x7fb40c004c80 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:34.350 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.346+0000 7fb42b579700 1 -- 172.21.15.27:0/2632019422 >> 172.21.15.27:0/2632019422 conn(0x7fb4240d0f70 msgr2=0x7fb4240ccf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:34.351 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.346+0000 7fb42b579700 1 -- 172.21.15.27:0/2632019422 shutdown_connections 2023-10-06T01:42:34.351 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.346+0000 7fb42b579700 1 -- 172.21.15.27:0/2632019422 wait complete. 2023-10-06T01:42:34.351 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.347+0000 7fb42b579700 1 Processor -- start 2023-10-06T01:42:34.351 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.347+0000 7fb42b579700 1 -- start start 2023-10-06T01:42:34.353 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.348+0000 7fb42b579700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb40c004810 0x7fb4240c77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:34.353 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.348+0000 7fb42b579700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb424169ad0 0x7fb4240c8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:34.353 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.348+0000 7fb42b579700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb4240ca370 0x7fb4240ca7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:34.353 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.348+0000 7fb42b579700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4240c7d20 con 0x7fb40c004810 2023-10-06T01:42:34.353 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.348+0000 7fb42b579700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb42416ab30 con 0x7fb4240ca370 2023-10-06T01:42:34.354 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.348+0000 7fb42b579700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4240b34d0 con 0x7fb424169ad0 2023-10-06T01:42:34.354 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.348+0000 7fb42a577700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb40c004810 0x7fb4240c77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:34.354 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.348+0000 7fb42a577700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb40c004810 0x7fb4240c77e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48386/0 (socket says 172.21.15.27:48386) 2023-10-06T01:42:34.354 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.348+0000 7fb42a577700 1 -- 172.21.15.27:0/2779294755 learned_addr learned my addr 172.21.15.27:0/2779294755 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:34.354 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.348+0000 7fb429d76700 1 --2- 172.21.15.27:0/2779294755 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb424169ad0 0x7fb4240c8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:34.354 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.348+0000 7fb42ad78700 1 --2- 172.21.15.27:0/2779294755 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb4240ca370 0x7fb4240ca7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:34.354 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.348+0000 7fb42a577700 1 -- 172.21.15.27:0/2779294755 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb424169ad0 msgr2=0x7fb4240c8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:34.354 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.348+0000 7fb42a577700 1 --2- 172.21.15.27:0/2779294755 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb424169ad0 0x7fb4240c8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:34.354 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.348+0000 7fb42a577700 1 -- 172.21.15.27:0/2779294755 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb4240ca370 msgr2=0x7fb4240ca7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:34.354 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.348+0000 7fb42a577700 1 --2- 172.21.15.27:0/2779294755 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb4240ca370 0x7fb4240ca7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:34.354 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.348+0000 7fb42a577700 1 -- 172.21.15.27:0/2779294755 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb42416a290 con 0x7fb40c004810 2023-10-06T01:42:34.355 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.348+0000 7fb42ad78700 1 --2- 172.21.15.27:0/2779294755 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb4240ca370 0x7fb4240ca7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:42:34.355 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.348+0000 7fb42a577700 1 --2- 172.21.15.27:0/2779294755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb40c004810 0x7fb4240c77e0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fb414007ac0 tx=0x7fb4140021d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:34.355 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.348+0000 7fb4227fc700 1 -- 172.21.15.27:0/2779294755 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41400ad10 con 0x7fb40c004810 2023-10-06T01:42:34.355 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.349+0000 7fb4227fc700 1 -- 172.21.15.27:0/2779294755 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb41400ae70 con 0x7fb40c004810 2023-10-06T01:42:34.355 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.349+0000 7fb42b579700 1 -- 172.21.15.27:0/2779294755 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb42416a570 con 0x7fb40c004810 2023-10-06T01:42:34.355 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.349+0000 7fb4227fc700 1 -- 172.21.15.27:0/2779294755 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb414013620 con 0x7fb40c004810 2023-10-06T01:42:34.355 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.349+0000 7fb42b579700 1 -- 172.21.15.27:0/2779294755 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4240c8e90 con 0x7fb40c004810 2023-10-06T01:42:34.355 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.350+0000 7fb42b579700 1 -- 172.21.15.27:0/2779294755 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb42404c3c0 con 0x7fb40c004810 2023-10-06T01:42:34.355 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.350+0000 7fb4227fc700 1 -- 172.21.15.27:0/2779294755 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb4140139d0 con 0x7fb40c004810 2023-10-06T01:42:34.355 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.350+0000 7fb4227fc700 1 --2- 172.21.15.27:0/2779294755 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb410066f30 0x7fb4100693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:34.355 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.351+0000 7fb4227fc700 1 -- 172.21.15.27:0/2779294755 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5292+0+0 (secure 0 0 0) 0x7fb414085120 con 0x7fb40c004810 2023-10-06T01:42:34.356 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.351+0000 7fb429d76700 1 --2- 172.21.15.27:0/2779294755 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb410066f30 0x7fb4100693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:34.356 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.351+0000 7fb429d76700 1 --2- 172.21.15.27:0/2779294755 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb410066f30 0x7fb4100693e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb4180091e0 tx=0x7fb41801a000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:34.356 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.355+0000 7fb4227fc700 1 -- 172.21.15.27:0/2779294755 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb414052530 con 0x7fb40c004810 2023-10-06T01:42:34.544 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.544+0000 7fb42b579700 1 -- 172.21.15.27:0/2779294755 --> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb4240b3610 con 0x7fb410066f30 2023-10-06T01:42:34.547 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.547+0000 7fb4227fc700 1 -- 172.21.15.27:0/2779294755 <== mgr.4100 v2:172.21.15.124:6832/90278 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+73220 (secure 0 0 0) 0x7fb418023040 con 0x7fb410066f30 2023-10-06T01:42:34.548 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:34.550 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.549+0000 7fb42b579700 1 -- 172.21.15.27:0/2779294755 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb410066f30 msgr2=0x7fb4100693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:34.550 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.549+0000 7fb42b579700 1 --2- 172.21.15.27:0/2779294755 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb410066f30 0x7fb4100693e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb4180091e0 tx=0x7fb41801a000 comp rx=0 tx=0).stop 2023-10-06T01:42:34.550 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.549+0000 7fb42b579700 1 -- 172.21.15.27:0/2779294755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb40c004810 msgr2=0x7fb4240c77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:34.550 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.549+0000 7fb42b579700 1 --2- 172.21.15.27:0/2779294755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb40c004810 0x7fb4240c77e0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fb414007ac0 tx=0x7fb4140021d0 comp rx=0 tx=0).stop 2023-10-06T01:42:34.550 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.549+0000 7fb42b579700 1 -- 172.21.15.27:0/2779294755 shutdown_connections 2023-10-06T01:42:34.550 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.549+0000 7fb42b579700 1 --2- 172.21.15.27:0/2779294755 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb410066f30 0x7fb4100693e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:34.550 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.549+0000 7fb42b579700 1 --2- 172.21.15.27:0/2779294755 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb4240ca370 0x7fb4240ca7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:34.550 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.549+0000 7fb42b579700 1 --2- 172.21.15.27:0/2779294755 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb40c004810 0x7fb4240c77e0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:34.550 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.549+0000 7fb42b579700 1 --2- 172.21.15.27:0/2779294755 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb424169ad0 0x7fb4240c8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:34.550 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.549+0000 7fb42b579700 1 -- 172.21.15.27:0/2779294755 >> 172.21.15.27:0/2779294755 conn(0x7fb4240d0f70 msgr2=0x7fb4240c5330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:34.550 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.549+0000 7fb42b579700 1 -- 172.21.15.27:0/2779294755 shutdown_connections 2023-10-06T01:42:34.551 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.549+0000 7fb42b579700 1 -- 172.21.15.27:0/2779294755 wait complete. 2023-10-06T01:42:34.551 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-06T01:42:34.561 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":75,"stamp":"2023-10-06T01:42:34.057618+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":143,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2382872,"kb_used_data":4568,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752591848,"statfs":{"total":773094113280,"available":770654052352,"internally_reserved":0,"allocated":4677632,"data_stored":2655194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-58022,"num_objects":-33,"num_object_clones":0,"num_object_copies":-66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-33,"num_whiteouts":0,"num_read":-22,"num_read_kb":-11,"num_write":-187,"num_write_kb":-131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-19,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001454"},"pg_stats":[{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-06T01:42:29.422519+0000","last_change":"2023-10-06T01:41:07.454295+0000","last_active":"2023-10-06T01:42:29.422519+0000","last_peered":"2023-10-06T01:42:29.422519+0000","last_clean":"2023-10-06T01:42:29.422519+0000","last_became_active":"2023-10-06T01:41:07.454034+0000","last_became_peered":"2023-10-06T01:41:07.454034+0000","last_unstale":"2023-10-06T01:42:29.422519+0000","last_undegraded":"2023-10-06T01:42:29.422519+0000","last_fullsized":"2023-10-06T01:42:29.422519+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:32:58.914183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":30,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-06T01:42:28.418774+0000","last_change":"2023-10-06T01:41:07.261955+0000","last_active":"2023-10-06T01:42:28.418774+0000","last_peered":"2023-10-06T01:42:28.418774+0000","last_clean":"2023-10-06T01:42:28.418774+0000","last_became_active":"2023-10-06T01:41:07.261692+0000","last_became_peered":"2023-10-06T01:41:07.261692+0000","last_unstale":"2023-10-06T01:42:28.418774+0000","last_undegraded":"2023-10-06T01:42:28.418774+0000","last_fullsized":"2023-10-06T01:42:28.418774+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":27,"reported_epoch":36,"state":"active+clean","last_fresh":"2023-10-06T01:42:28.773478+0000","last_change":"2023-10-06T01:41:07.258362+0000","last_active":"2023-10-06T01:42:28.773478+0000","last_peered":"2023-10-06T01:42:28.773478+0000","last_clean":"2023-10-06T01:42:28.773478+0000","last_became_active":"2023-10-06T01:41:07.258055+0000","last_became_peered":"2023-10-06T01:41:07.258055+0000","last_unstale":"2023-10-06T01:42:28.773478+0000","last_undegraded":"2023-10-06T01:42:28.773478+0000","last_fullsized":"2023-10-06T01:42:28.773478+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"16'117","reported_seq":178,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-06T01:42:29.513225+0000","last_change":"2023-10-06T01:41:07.261164+0000","last_active":"2023-10-06T01:42:29.513225+0000","last_peered":"2023-10-06T01:42:29.513225+0000","last_clean":"2023-10-06T01:42:29.513225+0000","last_became_active":"2023-10-06T01:41:07.260896+0000","last_became_peered":"2023-10-06T01:41:07.260896+0000","last_unstale":"2023-10-06T01:42:29.513225+0000","last_undegraded":"2023-10-06T01:42:29.513225+0000","last_fullsized":"2023-10-06T01:42:29.513225+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:05:22.934401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":28,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-06T01:42:29.515247+0000","last_change":"2023-10-06T01:41:07.456511+0000","last_active":"2023-10-06T01:42:29.515247+0000","last_peered":"2023-10-06T01:42:29.515247+0000","last_clean":"2023-10-06T01:42:29.515247+0000","last_became_active":"2023-10-06T01:41:07.456024+0000","last_became_peered":"2023-10-06T01:41:07.456024+0000","last_unstale":"2023-10-06T01:42:29.515247+0000","last_undegraded":"2023-10-06T01:42:29.515247+0000","last_fullsized":"2023-10-06T01:42:29.515247+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:51:02.949231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":28,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-06T01:42:29.516306+0000","last_change":"2023-10-06T01:41:07.456273+0000","last_active":"2023-10-06T01:42:29.516306+0000","last_peered":"2023-10-06T01:42:29.516306+0000","last_clean":"2023-10-06T01:42:29.516306+0000","last_became_active":"2023-10-06T01:41:07.456000+0000","last_became_peered":"2023-10-06T01:41:07.456000+0000","last_unstale":"2023-10-06T01:42:29.516306+0000","last_undegraded":"2023-10-06T01:42:29.516306+0000","last_fullsized":"2023-10-06T01:42:29.516306+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:01:42.633388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":29,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-06T01:42:29.518007+0000","last_change":"2023-10-06T01:41:07.456688+0000","last_active":"2023-10-06T01:42:29.518007+0000","last_peered":"2023-10-06T01:42:29.518007+0000","last_clean":"2023-10-06T01:42:29.518007+0000","last_became_active":"2023-10-06T01:41:07.456304+0000","last_became_peered":"2023-10-06T01:41:07.456304+0000","last_unstale":"2023-10-06T01:42:29.518007+0000","last_undegraded":"2023-10-06T01:42:29.518007+0000","last_fullsized":"2023-10-06T01:42:29.518007+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":30,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-06T01:42:29.516302+0000","last_change":"2023-10-06T01:41:07.456388+0000","last_active":"2023-10-06T01:42:29.516302+0000","last_peered":"2023-10-06T01:42:29.516302+0000","last_clean":"2023-10-06T01:42:29.516302+0000","last_became_active":"2023-10-06T01:41:07.456045+0000","last_became_peered":"2023-10-06T01:41:07.456045+0000","last_unstale":"2023-10-06T01:42:29.516302+0000","last_undegraded":"2023-10-06T01:42:29.516302+0000","last_fullsized":"2023-10-06T01:42:29.516302+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":30,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-06T01:42:29.517621+0000","last_change":"2023-10-06T01:41:07.258162+0000","last_active":"2023-10-06T01:42:29.517621+0000","last_peered":"2023-10-06T01:42:29.517621+0000","last_clean":"2023-10-06T01:42:29.517621+0000","last_became_active":"2023-10-06T01:41:07.257795+0000","last_became_peered":"2023-10-06T01:41:07.257795+0000","last_unstale":"2023-10-06T01:42:29.517621+0000","last_undegraded":"2023-10-06T01:42:29.517621+0000","last_fullsized":"2023-10-06T01:42:29.517621+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:05:22.934401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574871,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298304,"kb_used_data":1016,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073536,"statfs":{"total":96636764160,"available":96331300864,"internally_reserved":0,"allocated":1040384,"data_stored":778347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 01:42:08 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.51000000000000001}]},{"osd":1,"last update":"Fri Oct 6 01:42:08 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.49299999999999999}]},{"osd":2,"last update":"Fri Oct 6 01:42:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.622}]},{"osd":3,"last update":"Fri Oct 6 01:42:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.55000000000000004}]},{"osd":4,"last update":"Fri Oct 6 01:42:08 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.65000000000000002}]},{"osd":5,"last update":"Fri Oct 6 01:42:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.60599999999999998}]},{"osd":6,"last update":"Fri Oct 6 01:42:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.58399999999999996}]}]},{"osd":6,"up_from":13,"seq":55834574871,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297720,"kb_used_data":432,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074120,"statfs":{"total":96636764160,"available":96331898880,"internally_reserved":0,"allocated":442368,"data_stored":186626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 01:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.63900000000000001}]},{"osd":1,"last update":"Fri Oct 6 01:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.628}]},{"osd":2,"last update":"Fri Oct 6 01:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.78500000000000003}]},{"osd":3,"last update":"Fri Oct 6 01:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.66200000000000003}]},{"osd":4,"last update":"Fri Oct 6 01:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.51700000000000002}]},{"osd":5,"last update":"Fri Oct 6 01:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.501}]},{"osd":7,"last update":"Fri Oct 6 01:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.442},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.55200000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298288,"kb_used_data":1000,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073552,"statfs":{"total":96636764160,"available":96331317248,"internally_reserved":0,"allocated":1024000,"data_stored":772460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.255},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.27400000000000002}]},{"osd":2,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.318}]},{"osd":3,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.34100000000000003}]},{"osd":4,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.56899999999999995}]},{"osd":5,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.496}]},{"osd":6,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.59399999999999997}]},{"osd":7,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.55400000000000005}]}]},{"osd":0,"up_from":13,"seq":55834574869,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297688,"kb_used_data":400,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074152,"statfs":{"total":96636764160,"available":96331931648,"internally_reserved":0,"allocated":409600,"data_stored":174438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.495,"5min":0.495,"15min":0.495},"last":0.318},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"last":0.215}]},{"osd":2,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"last":0.29699999999999999}]},{"osd":3,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.439,"5min":0.439,"15min":0.439},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.34799999999999998}]},{"osd":4,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.56899999999999995}]},{"osd":5,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.55500000000000005}]},{"osd":6,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.505}]},{"osd":7,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.49099999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574870,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297724,"kb_used_data":436,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074116,"statfs":{"total":96636764160,"available":96331894784,"internally_reserved":0,"allocated":446464,"data_stored":187979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.497,"5min":0.497,"15min":0.497},"last":0.26700000000000002}]},{"osd":1,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.35499999999999998}]},{"osd":3,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.216},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.313}]},{"osd":4,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.56999999999999995}]},{"osd":5,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.63200000000000001}]},{"osd":6,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.51400000000000001}]},{"osd":7,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.502}]}]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297704,"kb_used_data":416,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074136,"statfs":{"total":96636764160,"available":96331915264,"internally_reserved":0,"allocated":425984,"data_stored":180739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.31,"15min":0.31},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"last":0.39700000000000002}]},{"osd":1,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.42999999999999999}]},{"osd":2,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"last":0.48799999999999999}]},{"osd":4,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.64100000000000001}]},{"osd":5,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.68000000000000005}]},{"osd":6,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.72199999999999998}]},{"osd":7,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.69099999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574870,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297724,"kb_used_data":436,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074116,"statfs":{"total":96636764160,"available":96331894784,"internally_reserved":0,"allocated":446464,"data_stored":187979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.61399999999999999}]},{"osd":1,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.58799999999999997}]},{"osd":2,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.52700000000000002}]},{"osd":3,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.75}]},{"osd":5,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.46600000000000003}]},{"osd":6,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.77900000000000003}]},{"osd":7,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.55000000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297720,"kb_used_data":432,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074120,"statfs":{"total":96636764160,"available":96331898880,"internally_reserved":0,"allocated":442368,"data_stored":186626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.335,"5min":1.335,"15min":1.335},"last":0.437}]},{"osd":1,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":2.4849999999999999,"5min":2.4849999999999999,"15min":2.4849999999999999},"last":0.55900000000000005}]},{"osd":2,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.8280000000000001,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":0.45900000000000002}]},{"osd":3,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.9830000000000001,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":0.5},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.36799999999999999}]},{"osd":4,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.45100000000000001}]},{"osd":6,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.34599999999999997}]},{"osd":7,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.47099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-06T01:42:34.564 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-06T01:42:34.682 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.682+0000 7f923b193700 1 Processor -- start 2023-10-06T01:42:34.683 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.682+0000 7f923b193700 1 -- start start 2023-10-06T01:42:34.685 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.683+0000 7f923b193700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f92340cbda0 0x7f9234067630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:34.685 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.683+0000 7f923b193700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9234067b70 0x7f923415c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:34.685 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.683+0000 7f923b193700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92340af040 con 0x7f92340cc170 2023-10-06T01:42:34.685 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.683+0000 7f923b193700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92340afd70 con 0x7f9234067b70 2023-10-06T01:42:34.685 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.683+0000 7f923b193700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9234068080 con 0x7f92340cbda0 2023-10-06T01:42:34.686 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.683+0000 7f9239990700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9234067b70 0x7f923415c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:34.686 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.683+0000 7f923a992700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f92340cc170 0x7f923415cc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:51318/0 (socket says 172.21.15.27:51318) 2023-10-06T01:42:34.686 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.683+0000 7f923a992700 1 -- 172.21.15.27:0/2213671223 learned_addr learned my addr 172.21.15.27:0/2213671223 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:34.686 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.683+0000 7f923a191700 1 --2- 172.21.15.27:0/2213671223 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f92340cbda0 0x7f9234067630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:34.686 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.683+0000 7f923918f700 1 -- 172.21.15.27:0/2213671223 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4274460588 0 0) 0x7f92340af040 con 0x7f92340cc170 2023-10-06T01:42:34.686 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.683+0000 7f923918f700 1 -- 172.21.15.27:0/2213671223 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f921c003570 con 0x7f92340cc170 2023-10-06T01:42:34.686 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.684+0000 7f9239990700 1 -- 172.21.15.27:0/2213671223 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f92340cbda0 msgr2=0x7f9234067630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:34.686 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.684+0000 7f9239990700 1 --2- 172.21.15.27:0/2213671223 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f92340cbda0 0x7f9234067630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:34.686 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.684+0000 7f9239990700 1 -- 172.21.15.27:0/2213671223 >> v1:172.21.15.27:6789/0 conn(0x7f92340cc170 legacy=0x7f923415cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:34.686 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.684+0000 7f9239990700 1 -- 172.21.15.27:0/2213671223 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92340af290 con 0x7f9234067b70 2023-10-06T01:42:34.686 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.684+0000 7f9239990700 1 --2- 172.21.15.27:0/2213671223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9234067b70 0x7f923415c720 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f92280098e0 tx=0x7f92280091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=611f5ace1e89556a server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:34.687 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.684+0000 7f923918f700 1 -- 172.21.15.27:0/2213671223 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9228011630 con 0x7f9234067b70 2023-10-06T01:42:34.687 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.684+0000 7f923918f700 1 -- 172.21.15.27:0/2213671223 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9228011790 con 0x7f9234067b70 2023-10-06T01:42:34.687 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.684+0000 7f923918f700 1 -- 172.21.15.27:0/2213671223 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9228011b30 con 0x7f9234067b70 2023-10-06T01:42:34.687 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.684+0000 7f923b193700 1 -- 172.21.15.27:0/2213671223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9234067b70 msgr2=0x7f923415c720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:34.687 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.684+0000 7f923b193700 1 --2- 172.21.15.27:0/2213671223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9234067b70 0x7f923415c720 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f92280098e0 tx=0x7f92280091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:34.687 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.684+0000 7f923b193700 1 -- 172.21.15.27:0/2213671223 shutdown_connections 2023-10-06T01:42:34.687 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.684+0000 7f923b193700 1 --2- 172.21.15.27:0/2213671223 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f92340cbda0 0x7f9234067630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:34.687 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.684+0000 7f923b193700 1 --2- 172.21.15.27:0/2213671223 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9234067b70 0x7f923415c720 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:34.687 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.684+0000 7f923b193700 1 -- 172.21.15.27:0/2213671223 >> 172.21.15.27:0/2213671223 conn(0x7f92340d0f70 msgr2=0x7f92340ccf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:34.687 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.685+0000 7f923b193700 1 -- 172.21.15.27:0/2213671223 shutdown_connections 2023-10-06T01:42:34.687 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.685+0000 7f923b193700 1 -- 172.21.15.27:0/2213671223 wait complete. 2023-10-06T01:42:34.690 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.686+0000 7f923b193700 1 Processor -- start 2023-10-06T01:42:34.690 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.686+0000 7f923b193700 1 -- start start 2023-10-06T01:42:34.690 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.686+0000 7f923b193700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9234067b70 0x7f923415e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:34.690 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.686+0000 7f923b193700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f92340cbda0 0x7f92341585b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:34.690 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.686+0000 7f923b193700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92340cc170 0x7f9234158af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:34.690 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.686+0000 7f923b193700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f923415e190 con 0x7f92340cc170 2023-10-06T01:42:34.690 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.686+0000 7f923b193700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f923415d580 con 0x7f9234067b70 2023-10-06T01:42:34.690 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.686+0000 7f923b193700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f923415d860 con 0x7f92340cbda0 2023-10-06T01:42:34.691 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.687+0000 7f923a992700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92340cc170 0x7f9234158af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:34.691 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.687+0000 7f923a992700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92340cc170 0x7f9234158af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:48410/0 (socket says 172.21.15.27:48410) 2023-10-06T01:42:34.691 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.687+0000 7f923a992700 1 -- 172.21.15.27:0/4189726689 learned_addr learned my addr 172.21.15.27:0/4189726689 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:34.691 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.687+0000 7f9239990700 1 --2- 172.21.15.27:0/4189726689 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f92340cbda0 0x7f92341585b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:34.691 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.687+0000 7f923a191700 1 --2- 172.21.15.27:0/4189726689 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9234067b70 0x7f923415e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:34.691 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.687+0000 7f923a992700 1 -- 172.21.15.27:0/4189726689 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f92340cbda0 msgr2=0x7f92341585b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:34.691 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.687+0000 7f923a992700 1 --2- 172.21.15.27:0/4189726689 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f92340cbda0 0x7f92341585b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:34.691 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.687+0000 7f923a992700 1 -- 172.21.15.27:0/4189726689 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9234067b70 msgr2=0x7f923415e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:34.691 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.687+0000 7f923a992700 1 --2- 172.21.15.27:0/4189726689 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9234067b70 0x7f923415e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:34.691 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.687+0000 7f923a992700 1 -- 172.21.15.27:0/4189726689 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9228009420 con 0x7f92340cc170 2023-10-06T01:42:34.691 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.687+0000 7f923a992700 1 --2- 172.21.15.27:0/4189726689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92340cc170 0x7f9234158af0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f9224001460 tx=0x7f9224001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:34.691 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.687+0000 7f92327fc700 1 -- 172.21.15.27:0/4189726689 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922400a870 con 0x7f92340cc170 2023-10-06T01:42:34.692 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.687+0000 7f92327fc700 1 -- 172.21.15.27:0/4189726689 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f922400a9d0 con 0x7f92340cc170 2023-10-06T01:42:34.692 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.687+0000 7f923b193700 1 -- 172.21.15.27:0/4189726689 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f923415db40 con 0x7f92340cc170 2023-10-06T01:42:34.692 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.687+0000 7f92327fc700 1 -- 172.21.15.27:0/4189726689 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922400ad50 con 0x7f92340cc170 2023-10-06T01:42:34.692 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.687+0000 7f923b193700 1 -- 172.21.15.27:0/4189726689 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9234156180 con 0x7f92340cc170 2023-10-06T01:42:34.692 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.688+0000 7f923b193700 1 -- 172.21.15.27:0/4189726689 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f923404c3c0 con 0x7f92340cc170 2023-10-06T01:42:34.692 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.689+0000 7f92327fc700 1 -- 172.21.15.27:0/4189726689 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9224007300 con 0x7f92340cc170 2023-10-06T01:42:34.692 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.689+0000 7f92327fc700 1 --2- 172.21.15.27:0/4189726689 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9220066f30 0x7f92200693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:34.692 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.689+0000 7f92327fc700 1 -- 172.21.15.27:0/4189726689 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5292+0+0 (secure 0 0 0) 0x7f9224084760 con 0x7f92340cc170 2023-10-06T01:42:34.692 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.689+0000 7f923a191700 1 --2- 172.21.15.27:0/4189726689 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9220066f30 0x7f92200693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:34.692 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.690+0000 7f923a191700 1 --2- 172.21.15.27:0/4189726689 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9220066f30 0x7f92200693e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f922c009db0 tx=0x7f922c0091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:34.693 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.693+0000 7f92327fc700 1 -- 172.21.15.27:0/4189726689 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9224051c10 con 0x7f92340cc170 2023-10-06T01:42:34.889 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.889+0000 7f923b193700 1 -- 172.21.15.27:0/4189726689 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f923415e2d0 con 0x7f92340cc170 2023-10-06T01:42:34.890 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.889+0000 7f92327fc700 1 -- 172.21.15.27:0/4189726689 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+13779 (secure 0 0 0) 0x7f9224055230 con 0x7f92340cc170 2023-10-06T01:42:34.890 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:34.890 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":38,"fsid":"20a7bff0-74f8-4f2f-b6c6-899e804bc965","created":"2023-10-06T01:40:54.474215+0000","modified":"2023-10-06T01:42:30.427970+0000","last_up_change":"2023-10-06T01:41:05.242847+0000","last_in_change":"2023-10-06T01:40:56.890960+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-10-06T01:41:05.685386+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-10-06T01:41:06.092995+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"b902c86d-800c-40c5-b152-b545e50d02c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6800","nonce":90644},{"type":"v1","addr":"172.21.15.27:6801","nonce":90644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6802","nonce":90644},{"type":"v1","addr":"172.21.15.27:6803","nonce":90644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6806","nonce":90644},{"type":"v1","addr":"172.21.15.27:6807","nonce":90644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6804","nonce":90644},{"type":"v1","addr":"172.21.15.27:6805","nonce":90644}]},"public_addr":"172.21.15.27:6801/90644","cluster_addr":"172.21.15.27:6803/90644","heartbeat_back_addr":"172.21.15.27:6807/90644","heartbeat_front_addr":"172.21.15.27:6805/90644","state":["exists","up"]},{"osd":1,"uuid":"5b4a70e5-b06e-48cd-bfc4-7962c5d05386","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6808","nonce":90643},{"type":"v1","addr":"172.21.15.27:6809","nonce":90643}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6810","nonce":90643},{"type":"v1","addr":"172.21.15.27:6811","nonce":90643}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6814","nonce":90643},{"type":"v1","addr":"172.21.15.27:6815","nonce":90643}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6812","nonce":90643},{"type":"v1","addr":"172.21.15.27:6813","nonce":90643}]},"public_addr":"172.21.15.27:6809/90643","cluster_addr":"172.21.15.27:6811/90643","heartbeat_back_addr":"172.21.15.27:6815/90643","heartbeat_front_addr":"172.21.15.27:6813/90643","state":["exists","up"]},{"osd":2,"uuid":"22d9de46-4073-4336-9122-5227e601eca0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6816","nonce":90645},{"type":"v1","addr":"172.21.15.27:6817","nonce":90645}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6818","nonce":90645},{"type":"v1","addr":"172.21.15.27:6819","nonce":90645}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6822","nonce":90645},{"type":"v1","addr":"172.21.15.27:6823","nonce":90645}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6820","nonce":90645},{"type":"v1","addr":"172.21.15.27:6821","nonce":90645}]},"public_addr":"172.21.15.27:6817/90645","cluster_addr":"172.21.15.27:6819/90645","heartbeat_back_addr":"172.21.15.27:6823/90645","heartbeat_front_addr":"172.21.15.27:6821/90645","state":["exists","up"]},{"osd":3,"uuid":"438bb2d3-34e2-4b7c-9734-8360c1b67650","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6824","nonce":90668},{"type":"v1","addr":"172.21.15.27:6825","nonce":90668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6826","nonce":90668},{"type":"v1","addr":"172.21.15.27:6827","nonce":90668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6830","nonce":90668},{"type":"v1","addr":"172.21.15.27:6831","nonce":90668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6828","nonce":90668},{"type":"v1","addr":"172.21.15.27:6829","nonce":90668}]},"public_addr":"172.21.15.27:6825/90668","cluster_addr":"172.21.15.27:6827/90668","heartbeat_back_addr":"172.21.15.27:6831/90668","heartbeat_front_addr":"172.21.15.27:6829/90668","state":["exists","up"]},{"osd":4,"uuid":"e1956a8e-e0f8-4e7c-acbb-40037415a354","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6800","nonce":90863},{"type":"v1","addr":"172.21.15.124:6801","nonce":90863}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6802","nonce":90863},{"type":"v1","addr":"172.21.15.124:6803","nonce":90863}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6806","nonce":90863},{"type":"v1","addr":"172.21.15.124:6807","nonce":90863}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6804","nonce":90863},{"type":"v1","addr":"172.21.15.124:6805","nonce":90863}]},"public_addr":"172.21.15.124:6801/90863","cluster_addr":"172.21.15.124:6803/90863","heartbeat_back_addr":"172.21.15.124:6807/90863","heartbeat_front_addr":"172.21.15.124:6805/90863","state":["exists","up"]},{"osd":5,"uuid":"306c59d4-846d-4b73-b1c9-051bec9e9368","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6816","nonce":90885},{"type":"v1","addr":"172.21.15.124:6817","nonce":90885}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6818","nonce":90885},{"type":"v1","addr":"172.21.15.124:6819","nonce":90885}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6822","nonce":90885},{"type":"v1","addr":"172.21.15.124:6823","nonce":90885}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6820","nonce":90885},{"type":"v1","addr":"172.21.15.124:6821","nonce":90885}]},"public_addr":"172.21.15.124:6817/90885","cluster_addr":"172.21.15.124:6819/90885","heartbeat_back_addr":"172.21.15.124:6823/90885","heartbeat_front_addr":"172.21.15.124:6821/90885","state":["exists","up"]},{"osd":6,"uuid":"15463e53-ac8d-47d1-b4f6-c0d31f29e4a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6808","nonce":90864},{"type":"v1","addr":"172.21.15.124:6809","nonce":90864}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6810","nonce":90864},{"type":"v1","addr":"172.21.15.124:6811","nonce":90864}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6814","nonce":90864},{"type":"v1","addr":"172.21.15.124:6815","nonce":90864}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6812","nonce":90864},{"type":"v1","addr":"172.21.15.124:6813","nonce":90864}]},"public_addr":"172.21.15.124:6809/90864","cluster_addr":"172.21.15.124:6811/90864","heartbeat_back_addr":"172.21.15.124:6815/90864","heartbeat_front_addr":"172.21.15.124:6813/90864","state":["exists","up"]},{"osd":7,"uuid":"ae463a90-a9d3-498f-86fa-47a35b3fe889","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6824","nonce":90894},{"type":"v1","addr":"172.21.15.124:6825","nonce":90894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6826","nonce":90894},{"type":"v1","addr":"172.21.15.124:6827","nonce":90894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6830","nonce":90894},{"type":"v1","addr":"172.21.15.124:6831","nonce":90894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6828","nonce":90894},{"type":"v1","addr":"172.21.15.124:6829","nonce":90894}]},"public_addr":"172.21.15.124:6825/90894","cluster_addr":"172.21.15.124:6827/90894","heartbeat_back_addr":"172.21.15.124:6831/90894","heartbeat_front_addr":"172.21.15.124:6829/90894","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.827704+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.839951+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.032230+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.050605+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:01.987908+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.031290+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.080707+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-10-06T01:41:02.053692+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.124:6837/325247112":"2023-10-07T01:42:26.110568+0000","172.21.15.124:6835/325247112":"2023-10-07T01:42:26.110568+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-06T01:42:34.892 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.892+0000 7f923b193700 1 -- 172.21.15.27:0/4189726689 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9220066f30 msgr2=0x7f92200693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:34.892 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.892+0000 7f923b193700 1 --2- 172.21.15.27:0/4189726689 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9220066f30 0x7f92200693e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f922c009db0 tx=0x7f922c0091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:34.892 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.892+0000 7f923b193700 1 -- 172.21.15.27:0/4189726689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92340cc170 msgr2=0x7f9234158af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:34.893 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.892+0000 7f923b193700 1 --2- 172.21.15.27:0/4189726689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92340cc170 0x7f9234158af0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f9224001460 tx=0x7f9224001a40 comp rx=0 tx=0).stop 2023-10-06T01:42:34.893 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.892+0000 7f923b193700 1 -- 172.21.15.27:0/4189726689 shutdown_connections 2023-10-06T01:42:34.893 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.892+0000 7f923b193700 1 --2- 172.21.15.27:0/4189726689 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9220066f30 0x7f92200693e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:34.893 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.892+0000 7f923b193700 1 --2- 172.21.15.27:0/4189726689 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f92340cc170 0x7f9234158af0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:34.893 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.892+0000 7f923b193700 1 --2- 172.21.15.27:0/4189726689 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9234067b70 0x7f923415e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:34.893 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.892+0000 7f923b193700 1 --2- 172.21.15.27:0/4189726689 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f92340cbda0 0x7f92341585b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:34.894 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.892+0000 7f923b193700 1 -- 172.21.15.27:0/4189726689 >> 172.21.15.27:0/4189726689 conn(0x7f92340d0f70 msgr2=0x7f923415b9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:34.894 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.892+0000 7f923b193700 1 -- 172.21.15.27:0/4189726689 shutdown_connections 2023-10-06T01:42:34.894 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:34.892+0000 7f923b193700 1 -- 172.21.15.27:0/4189726689 wait complete. 2023-10-06T01:42:35.905 INFO:tasks.ceph:Scrubbing osd.0 2023-10-06T01:42:35.905 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-06T01:42:36.033 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.033+0000 7fc4eb59e700 1 Processor -- start 2023-10-06T01:42:36.034 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.033+0000 7fc4eb59e700 1 -- start start 2023-10-06T01:42:36.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.034+0000 7fc4eb59e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc4ec0cf440 0x7fc4ec0cbb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:36.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.034+0000 7fc4eb59e700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fc4ec0cf810 0x7fc4ec0cc140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:36.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.034+0000 7fc4eb59e700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4ec0cb670 con 0x7fc4ec0cc680 2023-10-06T01:42:36.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.034+0000 7fc4eb59e700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4ec0cce90 con 0x7fc4ec0cf810 2023-10-06T01:42:36.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.034+0000 7fc4eb59e700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ec0cad80 con 0x7fc4ec0cf440 2023-10-06T01:42:36.037 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.034+0000 7fc4e9d9b700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fc4ec0cf810 0x7fc4ec0cc140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:36.037 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.034+0000 7fc4ea59c700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc4ec0cf440 0x7fc4ec0cbb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:36.037 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.034+0000 7fc4ea59c700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc4ec0cf440 0x7fc4ec0cbb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.27:44926/0 (socket says 172.21.15.27:44926) 2023-10-06T01:42:36.037 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.034+0000 7fc4e9d9b700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fc4ec0cf810 0x7fc4ec0cc140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3300/0 says I am v2:172.21.15.27:42138/0 (socket says 172.21.15.27:42138) 2023-10-06T01:42:36.037 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.034+0000 7fc4ea59c700 1 -- 172.21.15.27:0/4121756334 learned_addr learned my addr 172.21.15.27:0/4121756334 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:36.037 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.034+0000 7fc4ead9d700 1 --1- 172.21.15.27:0/4121756334 >> v1:172.21.15.124:6789/0 conn(0x7fc4ec0cc680 0x7fc4ec15d450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.124:6789/0 says I am v1:172.21.15.27:48070/0 (socket says 172.21.15.27:48070) 2023-10-06T01:42:36.037 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.035+0000 7fc4e959a700 1 -- 172.21.15.27:0/4121756334 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2027368845 0 0) 0x7fc4ec0cb670 con 0x7fc4ec0cc680 2023-10-06T01:42:36.037 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.035+0000 7fc4e9d9b700 1 -- 172.21.15.27:0/4121756334 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc4ec0cf440 msgr2=0x7fc4ec0cbb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:36.037 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.035+0000 7fc4e9d9b700 1 --2- 172.21.15.27:0/4121756334 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc4ec0cf440 0x7fc4ec0cbb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:36.037 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.035+0000 7fc4e9d9b700 1 -- 172.21.15.27:0/4121756334 >> v1:172.21.15.124:6789/0 conn(0x7fc4ec0cc680 legacy=0x7fc4ec15d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:36.037 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.035+0000 7fc4e9d9b700 1 -- 172.21.15.27:0/4121756334 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4ec0cb020 con 0x7fc4ec0cf810 2023-10-06T01:42:36.037 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.035+0000 7fc4ea59c700 1 --2- 172.21.15.27:0/4121756334 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc4ec0cf440 0x7fc4ec0cbb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:42:36.038 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.035+0000 7fc4e9d9b700 1 --2- 172.21.15.27:0/4121756334 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc4ec0cf810 0x7fc4ec0cc140 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc4d40098e0 tx=0x7fc4d40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa5001ea2c1e0787 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:36.038 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.035+0000 7fc4e959a700 1 -- 172.21.15.27:0/4121756334 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d400b640 con 0x7fc4ec0cf810 2023-10-06T01:42:36.038 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.035+0000 7fc4e959a700 1 -- 172.21.15.27:0/4121756334 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4d400b7a0 con 0x7fc4ec0cf810 2023-10-06T01:42:36.038 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.035+0000 7fc4e959a700 1 -- 172.21.15.27:0/4121756334 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d400bb40 con 0x7fc4ec0cf810 2023-10-06T01:42:36.038 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.036+0000 7fc4eb59e700 1 -- 172.21.15.27:0/4121756334 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc4ec0cf810 msgr2=0x7fc4ec0cc140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:36.038 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.036+0000 7fc4eb59e700 1 --2- 172.21.15.27:0/4121756334 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc4ec0cf810 0x7fc4ec0cc140 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc4d40098e0 tx=0x7fc4d40091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:36.038 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.036+0000 7fc4eb59e700 1 -- 172.21.15.27:0/4121756334 shutdown_connections 2023-10-06T01:42:36.038 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.036+0000 7fc4eb59e700 1 --2- 172.21.15.27:0/4121756334 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc4ec0cf440 0x7fc4ec0cbb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:36.038 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.036+0000 7fc4eb59e700 1 --2- 172.21.15.27:0/4121756334 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc4ec0cf810 0x7fc4ec0cc140 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:36.038 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.036+0000 7fc4eb59e700 1 -- 172.21.15.27:0/4121756334 >> 172.21.15.27:0/4121756334 conn(0x7fc4ec0cd080 msgr2=0x7fc4ec0cd480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:36.038 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.036+0000 7fc4eb59e700 1 -- 172.21.15.27:0/4121756334 shutdown_connections 2023-10-06T01:42:36.038 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.036+0000 7fc4eb59e700 1 -- 172.21.15.27:0/4121756334 wait complete. 2023-10-06T01:42:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.037+0000 7fc4eb59e700 1 Processor -- start 2023-10-06T01:42:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.037+0000 7fc4eb59e700 1 -- start start 2023-10-06T01:42:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.038+0000 7fc4eb59e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4ec0cc680 0x7fc4ec16a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.038+0000 7fc4eb59e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc4ec0cf440 0x7fc4ec14d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.038+0000 7fc4eb59e700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc4ec0cf810 0x7fc4ec169440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.038+0000 7fc4eb59e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ec16ab50 con 0x7fc4ec0cc680 2023-10-06T01:42:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.038+0000 7fc4eb59e700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ec15a400 con 0x7fc4ec0cf810 2023-10-06T01:42:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.038+0000 7fc4eb59e700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4ec169f00 con 0x7fc4ec0cf440 2023-10-06T01:42:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.038+0000 7fc4ea59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4ec0cc680 0x7fc4ec16a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.038+0000 7fc4ea59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4ec0cc680 0x7fc4ec16a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45772/0 (socket says 172.21.15.27:45772) 2023-10-06T01:42:36.042 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.038+0000 7fc4ea59c700 1 -- 172.21.15.27:0/3113487178 learned_addr learned my addr 172.21.15.27:0/3113487178 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.038+0000 7fc4e9d9b700 1 --2- 172.21.15.27:0/3113487178 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc4ec0cf440 0x7fc4ec14d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.038+0000 7fc4ead9d700 1 --2- 172.21.15.27:0/3113487178 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc4ec0cf810 0x7fc4ec169440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.038+0000 7fc4ea59c700 1 -- 172.21.15.27:0/3113487178 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc4ec0cf440 msgr2=0x7fc4ec14d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.038+0000 7fc4ea59c700 1 --2- 172.21.15.27:0/3113487178 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc4ec0cf440 0x7fc4ec14d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.038+0000 7fc4ea59c700 1 -- 172.21.15.27:0/3113487178 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc4ec0cf810 msgr2=0x7fc4ec169440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.038+0000 7fc4ea59c700 1 --2- 172.21.15.27:0/3113487178 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc4ec0cf810 0x7fc4ec169440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.038+0000 7fc4ea59c700 1 -- 172.21.15.27:0/3113487178 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4dc009ae0 con 0x7fc4ec0cc680 2023-10-06T01:42:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.038+0000 7fc4ead9d700 1 --2- 172.21.15.27:0/3113487178 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc4ec0cf810 0x7fc4ec169440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:42:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.038+0000 7fc4e9d9b700 1 --2- 172.21.15.27:0/3113487178 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc4ec0cf440 0x7fc4ec14d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:42:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.038+0000 7fc4ea59c700 1 --2- 172.21.15.27:0/3113487178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4ec0cc680 0x7fc4ec16a610 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc00c170 tx=0x7fc4dc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.038+0000 7fc4e27fc700 1 -- 172.21.15.27:0/3113487178 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc00caa0 con 0x7fc4ec0cc680 2023-10-06T01:42:36.043 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.038+0000 7fc4e27fc700 1 -- 172.21.15.27:0/3113487178 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4dc00f040 con 0x7fc4ec0cc680 2023-10-06T01:42:36.044 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.039+0000 7fc4e27fc700 1 -- 172.21.15.27:0/3113487178 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc00ce30 con 0x7fc4ec0cc680 2023-10-06T01:42:36.044 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.039+0000 7fc4eb59e700 1 -- 172.21.15.27:0/3113487178 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4d4009420 con 0x7fc4ec0cc680 2023-10-06T01:42:36.044 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.039+0000 7fc4eb59e700 1 -- 172.21.15.27:0/3113487178 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4ec15a090 con 0x7fc4ec0cc680 2023-10-06T01:42:36.044 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.039+0000 7fc4eb59e700 1 -- 172.21.15.27:0/3113487178 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc4ec159ef0 con 0x7fc4ec0cc680 2023-10-06T01:42:36.044 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.040+0000 7fc4e27fc700 1 -- 172.21.15.27:0/3113487178 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc4dc014690 con 0x7fc4ec0cc680 2023-10-06T01:42:36.044 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.040+0000 7fc4e27fc700 1 --2- 172.21.15.27:0/3113487178 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc4cc066f30 0x7fc4cc0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:36.044 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.040+0000 7fc4e27fc700 1 -- 172.21.15.27:0/3113487178 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5292+0+0 (secure 0 0 0) 0x7fc4dc085470 con 0x7fc4ec0cc680 2023-10-06T01:42:36.044 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.041+0000 7fc4e27fc700 1 --2- 172.21.15.27:0/3113487178 >> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] conn(0x7fc4cc06d3a0 0x7fc4cc06f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:36.044 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.041+0000 7fc4e27fc700 1 -- 172.21.15.27:0/3113487178 --> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc4cc06fe80 con 0x7fc4cc06d3a0 2023-10-06T01:42:36.044 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.041+0000 7fc4e27fc700 1 -- 172.21.15.27:0/3113487178 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc4dc085760 con 0x7fc4ec0cc680 2023-10-06T01:42:36.044 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.041+0000 7fc4e9d9b700 1 --2- 172.21.15.27:0/3113487178 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc4cc066f30 0x7fc4cc0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:36.044 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.041+0000 7fc4ead9d700 1 --2- 172.21.15.27:0/3113487178 >> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] conn(0x7fc4cc06d3a0 0x7fc4cc06f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:36.045 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.041+0000 7fc4ead9d700 1 --2- 172.21.15.27:0/3113487178 >> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] conn(0x7fc4cc06d3a0 0x7fc4cc06f7b0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:36.045 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.041+0000 7fc4e9d9b700 1 --2- 172.21.15.27:0/3113487178 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc4cc066f30 0x7fc4cc0693e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc4d40091e0 tx=0x7fc4d4017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:36.045 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.042+0000 7fc4e27fc700 1 -- 172.21.15.27:0/3113487178 <== osd.0 v2:172.21.15.27:6800/90644 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fc4cc06fe80 con 0x7fc4cc06d3a0 2023-10-06T01:42:36.055 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.055+0000 7fc4eb59e700 1 -- 172.21.15.27:0/3113487178 --> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fc4ec16a400 con 0x7fc4cc06d3a0 2023-10-06T01:42:36.056 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.056+0000 7fc4e27fc700 1 -- 172.21.15.27:0/3113487178 <== osd.0 v2:172.21.15.27:6800/90644 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fc4ec16a400 con 0x7fc4cc06d3a0 2023-10-06T01:42:36.056 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2023-10-06T01:42:36.056 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-06T01:42:36.056 INFO:teuthology.orchestra.run.smithi027.stdout:} 2023-10-06T01:42:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.056+0000 7fc4eb59e700 1 -- 172.21.15.27:0/3113487178 >> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] conn(0x7fc4cc06d3a0 msgr2=0x7fc4cc06f7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.056+0000 7fc4eb59e700 1 --2- 172.21.15.27:0/3113487178 >> [v2:172.21.15.27:6800/90644,v1:172.21.15.27:6801/90644] conn(0x7fc4cc06d3a0 0x7fc4cc06f7b0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.056+0000 7fc4eb59e700 1 -- 172.21.15.27:0/3113487178 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc4cc066f30 msgr2=0x7fc4cc0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.056+0000 7fc4eb59e700 1 --2- 172.21.15.27:0/3113487178 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc4cc066f30 0x7fc4cc0693e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc4d40091e0 tx=0x7fc4d4017000 comp rx=0 tx=0).stop 2023-10-06T01:42:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.056+0000 7fc4eb59e700 1 -- 172.21.15.27:0/3113487178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4ec0cc680 msgr2=0x7fc4ec16a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.056+0000 7fc4eb59e700 1 --2- 172.21.15.27:0/3113487178 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc4ec0cc680 0x7fc4ec16a610 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc00c170 tx=0x7fc4dc00b040 comp rx=0 tx=0).stop 2023-10-06T01:42:36.057 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.056+0000 7fc4ead9d700 1 -- 172.21.15.27:0/3113487178 reap_dead start 2023-10-06T01:42:36.058 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.056+0000 7fc4eb59e700 1 -- 172.21.15.27:0/3113487178 shutdown_connections 2023-10-06T01:42:36.058 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.056+0000 7fc4eb59e700 1 -- 172.21.15.27:0/3113487178 >> 172.21.15.27:0/3113487178 conn(0x7fc4ec0cd080 msgr2=0x7fc4ec157fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:36.058 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.056+0000 7fc4eb59e700 1 -- 172.21.15.27:0/3113487178 shutdown_connections 2023-10-06T01:42:36.058 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.056+0000 7fc4eb59e700 1 -- 172.21.15.27:0/3113487178 wait complete. 2023-10-06T01:42:36.068 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-06T01:42:36.187 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.187+0000 7f0c112ef700 1 Processor -- start 2023-10-06T01:42:36.187 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.187+0000 7f0c112ef700 1 -- start start 2023-10-06T01:42:36.189 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.188+0000 7f0c112ef700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0c0c1598d0 0x7f0c0c15d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:36.190 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.188+0000 7f0c112ef700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f0c0c15d8e0 0x7f0c0c0cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:36.190 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.188+0000 7f0c112ef700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c0c15dd20 con 0x7f0c0c159500 2023-10-06T01:42:36.190 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.188+0000 7f0c112ef700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c0c0b1110 con 0x7f0c0c15d8e0 2023-10-06T01:42:36.190 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.188+0000 7f0c112ef700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c0b13b0 con 0x7f0c0c1598d0 2023-10-06T01:42:36.190 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.188+0000 7f0c0affd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f0c0c159500 0x7f0c0c0b3fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47338/0 (socket says 172.21.15.27:47338) 2023-10-06T01:42:36.190 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.188+0000 7f0c0affd700 1 -- 172.21.15.27:0/2663311551 learned_addr learned my addr 172.21.15.27:0/2663311551 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:36.190 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.188+0000 7f0c0a7fc700 1 --2- 172.21.15.27:0/2663311551 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0c0c1598d0 0x7f0c0c15d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:36.190 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.188+0000 7f0c0b7fe700 1 --2- 172.21.15.27:0/2663311551 >> v2:172.21.15.124:3300/0 conn(0x7f0c0c15d8e0 0x7f0c0c0cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:36.190 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.188+0000 7f0c09ffb700 1 -- 172.21.15.27:0/2663311551 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 329518974 0 0) 0x7f0c0c15dd20 con 0x7f0c0c159500 2023-10-06T01:42:36.190 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.188+0000 7f0c09ffb700 1 -- 172.21.15.27:0/2663311551 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bf8003570 con 0x7f0c0c159500 2023-10-06T01:42:36.190 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.188+0000 7f0c0b7fe700 1 -- 172.21.15.27:0/2663311551 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0c0c1598d0 msgr2=0x7f0c0c15d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:36.191 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.188+0000 7f0c09ffb700 1 -- 172.21.15.27:0/2663311551 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2666273423 0 0) 0x7f0bf8003570 con 0x7f0c0c159500 2023-10-06T01:42:36.191 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.188+0000 7f0c0b7fe700 1 --2- 172.21.15.27:0/2663311551 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0c0c1598d0 0x7f0c0c15d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:36.191 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.188+0000 7f0c0b7fe700 1 -- 172.21.15.27:0/2663311551 >> v1:172.21.15.27:6789/0 conn(0x7f0c0c159500 legacy=0x7f0c0c0b3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:36.191 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.188+0000 7f0c0b7fe700 1 -- 172.21.15.27:0/2663311551 --> v2:172.21.15.124:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c0c0b1690 con 0x7f0c0c15d8e0 2023-10-06T01:42:36.191 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.188+0000 7f0c0a7fc700 1 --2- 172.21.15.27:0/2663311551 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0c0c1598d0 0x7f0c0c15d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:42:36.191 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.188+0000 7f0c0b7fe700 1 --2- 172.21.15.27:0/2663311551 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c0c15d8e0 0x7f0c0c0cf6c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc0098e0 tx=0x7f0bfc0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bae44591d5989f19 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:36.191 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.189+0000 7f0c09ffb700 1 -- 172.21.15.27:0/2663311551 <== mon.1 v2:172.21.15.124:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c040017f0 con 0x7f0c0c15d8e0 2023-10-06T01:42:36.191 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.189+0000 7f0c09ffb700 1 -- 172.21.15.27:0/2663311551 <== mon.1 v2:172.21.15.124:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bfc011420 con 0x7f0c0c15d8e0 2023-10-06T01:42:36.191 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.189+0000 7f0c09ffb700 1 -- 172.21.15.27:0/2663311551 <== mon.1 v2:172.21.15.124:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc0117c0 con 0x7f0c0c15d8e0 2023-10-06T01:42:36.191 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.189+0000 7f0c112ef700 1 -- 172.21.15.27:0/2663311551 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c0c15d8e0 msgr2=0x7f0c0c0cf6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:36.191 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.189+0000 7f0c112ef700 1 --2- 172.21.15.27:0/2663311551 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c0c15d8e0 0x7f0c0c0cf6c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc0098e0 tx=0x7f0bfc0091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:36.191 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.189+0000 7f0c112ef700 1 -- 172.21.15.27:0/2663311551 shutdown_connections 2023-10-06T01:42:36.192 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.189+0000 7f0c112ef700 1 --2- 172.21.15.27:0/2663311551 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c0c15d8e0 0x7f0c0c0cf6c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:36.192 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.189+0000 7f0c112ef700 1 --2- 172.21.15.27:0/2663311551 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0c0c1598d0 0x7f0c0c15d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:36.192 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.189+0000 7f0c112ef700 1 -- 172.21.15.27:0/2663311551 >> 172.21.15.27:0/2663311551 conn(0x7f0c0c067630 msgr2=0x7f0c0c069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:36.192 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.189+0000 7f0c112ef700 1 -- 172.21.15.27:0/2663311551 shutdown_connections 2023-10-06T01:42:36.192 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.189+0000 7f0c112ef700 1 -- 172.21.15.27:0/2663311551 wait complete. 2023-10-06T01:42:36.194 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.190+0000 7f0c112ef700 1 Processor -- start 2023-10-06T01:42:36.195 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.191+0000 7f0c112ef700 1 -- start start 2023-10-06T01:42:36.195 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.191+0000 7f0c112ef700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0c0c159500 0x7f0c0c150070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:36.195 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.191+0000 7f0c112ef700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c0c1598d0 0x7f0c0c14c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:36.195 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.191+0000 7f0c112ef700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c0c15d8e0 0x7f0c0c14d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:36.195 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.191+0000 7f0c112ef700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c151160 con 0x7f0c0c15d8e0 2023-10-06T01:42:36.195 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.191+0000 7f0c112ef700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c1508c0 con 0x7f0c0c1598d0 2023-10-06T01:42:36.195 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.191+0000 7f0c112ef700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c150ba0 con 0x7f0c0c159500 2023-10-06T01:42:36.195 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.191+0000 7f0c0b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c0c15d8e0 0x7f0c0c14d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:36.195 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.191+0000 7f0c0b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c0c15d8e0 0x7f0c0c14d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45786/0 (socket says 172.21.15.27:45786) 2023-10-06T01:42:36.195 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.191+0000 7f0c0b7fe700 1 -- 172.21.15.27:0/1142128025 learned_addr learned my addr 172.21.15.27:0/1142128025 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:36.196 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.192+0000 7f0c0affd700 1 --2- 172.21.15.27:0/1142128025 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0c0c159500 0x7f0c0c150070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:36.196 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.192+0000 7f0c0a7fc700 1 --2- 172.21.15.27:0/1142128025 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c0c1598d0 0x7f0c0c14c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:36.196 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.192+0000 7f0c0b7fe700 1 -- 172.21.15.27:0/1142128025 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0c0c159500 msgr2=0x7f0c0c150070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:36.196 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.192+0000 7f0c0b7fe700 1 --2- 172.21.15.27:0/1142128025 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0c0c159500 0x7f0c0c150070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:36.196 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.192+0000 7f0c0b7fe700 1 -- 172.21.15.27:0/1142128025 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c0c1598d0 msgr2=0x7f0c0c14c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:36.196 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.192+0000 7f0c0b7fe700 1 --2- 172.21.15.27:0/1142128025 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c0c1598d0 0x7f0c0c14c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:36.196 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.192+0000 7f0c0b7fe700 1 -- 172.21.15.27:0/1142128025 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf4009420 con 0x7f0c0c15d8e0 2023-10-06T01:42:36.196 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.192+0000 7f0c0affd700 1 --2- 172.21.15.27:0/1142128025 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0c0c159500 0x7f0c0c150070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:42:36.196 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.192+0000 7f0c0b7fe700 1 --2- 172.21.15.27:0/1142128025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c0c15d8e0 0x7f0c0c14d840 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c15deb0 tx=0x7f0bfc016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:36.196 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.192+0000 7f0c02ffd700 1 -- 172.21.15.27:0/1142128025 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc01cd60 con 0x7f0c0c15d8e0 2023-10-06T01:42:36.196 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.192+0000 7f0c02ffd700 1 -- 172.21.15.27:0/1142128025 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bfc01a450 con 0x7f0c0c15d8e0 2023-10-06T01:42:36.196 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.192+0000 7f0c02ffd700 1 -- 172.21.15.27:0/1142128025 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc01a810 con 0x7f0c0c15d8e0 2023-10-06T01:42:36.196 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.192+0000 7f0c112ef700 1 -- 172.21.15.27:0/1142128025 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bfc009420 con 0x7f0c0c15d8e0 2023-10-06T01:42:36.197 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.192+0000 7f0c112ef700 1 -- 172.21.15.27:0/1142128025 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c0c0b4170 con 0x7f0c0c15d8e0 2023-10-06T01:42:36.197 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.193+0000 7f0c112ef700 1 -- 172.21.15.27:0/1142128025 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c0c0b4390 con 0x7f0c0c15d8e0 2023-10-06T01:42:36.197 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.194+0000 7f0c02ffd700 1 -- 172.21.15.27:0/1142128025 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f0bfc01abc0 con 0x7f0c0c15d8e0 2023-10-06T01:42:36.197 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.194+0000 7f0c02ffd700 1 --2- 172.21.15.27:0/1142128025 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0bec066f30 0x7f0bec0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:36.197 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.194+0000 7f0c02ffd700 1 -- 172.21.15.27:0/1142128025 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5292+0+0 (secure 0 0 0) 0x7f0bfc085f70 con 0x7f0c0c15d8e0 2023-10-06T01:42:36.197 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.194+0000 7f0c0affd700 1 --2- 172.21.15.27:0/1142128025 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0bec066f30 0x7f0bec0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:36.197 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.195+0000 7f0c0affd700 1 --2- 172.21.15.27:0/1142128025 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0bec066f30 0x7f0bec0693e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0c04002250 tx=0x7f0c04009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:36.198 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.198+0000 7f0c02ffd700 1 -- 172.21.15.27:0/1142128025 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0bfc0532e0 con 0x7f0c0c15d8e0 2023-10-06T01:42:36.391 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.391+0000 7f0c112ef700 1 -- 172.21.15.27:0/1142128025 --> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f0c0c064050 con 0x7f0bec066f30 2023-10-06T01:42:36.392 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.392+0000 7f0c02ffd700 1 -- 172.21.15.27:0/1142128025 <== mgr.4100 v2:172.21.15.124:6832/90278 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0c0c064050 con 0x7f0bec066f30 2023-10-06T01:42:36.394 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.393+0000 7f0c112ef700 1 -- 172.21.15.27:0/1142128025 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0bec066f30 msgr2=0x7f0bec0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:36.394 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.393+0000 7f0c112ef700 1 --2- 172.21.15.27:0/1142128025 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0bec066f30 0x7f0bec0693e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0c04002250 tx=0x7f0c04009040 comp rx=0 tx=0).stop 2023-10-06T01:42:36.394 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.393+0000 7f0c112ef700 1 -- 172.21.15.27:0/1142128025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c0c15d8e0 msgr2=0x7f0c0c14d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:36.394 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.393+0000 7f0c112ef700 1 --2- 172.21.15.27:0/1142128025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c0c15d8e0 0x7f0c0c14d840 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c15deb0 tx=0x7f0bfc016000 comp rx=0 tx=0).stop 2023-10-06T01:42:36.394 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.393+0000 7f0c112ef700 1 -- 172.21.15.27:0/1142128025 shutdown_connections 2023-10-06T01:42:36.394 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.393+0000 7f0c112ef700 1 --2- 172.21.15.27:0/1142128025 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f0bec066f30 0x7f0bec0693e0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:36.395 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.393+0000 7f0c112ef700 1 --2- 172.21.15.27:0/1142128025 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f0c0c159500 0x7f0c0c150070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:36.395 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.393+0000 7f0c112ef700 1 --2- 172.21.15.27:0/1142128025 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f0c0c15d8e0 0x7f0c0c14d840 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:36.395 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.393+0000 7f0c112ef700 1 --2- 172.21.15.27:0/1142128025 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f0c0c1598d0 0x7f0c0c14c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:36.395 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.393+0000 7f0c112ef700 1 -- 172.21.15.27:0/1142128025 >> 172.21.15.27:0/1142128025 conn(0x7f0c0c067630 msgr2=0x7f0c0c156ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:36.395 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.393+0000 7f0c112ef700 1 -- 172.21.15.27:0/1142128025 shutdown_connections 2023-10-06T01:42:36.395 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.393+0000 7f0c112ef700 1 -- 172.21.15.27:0/1142128025 wait complete. 2023-10-06T01:42:36.396 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 0 to deep-scrub 2023-10-06T01:42:36.406 INFO:tasks.ceph:Scrubbing osd.1 2023-10-06T01:42:36.406 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-06T01:42:36.529 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.529+0000 7f4205f52700 1 Processor -- start 2023-10-06T01:42:36.529 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.529+0000 7f4205f52700 1 -- start start 2023-10-06T01:42:36.533 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.530+0000 7f4205f52700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f42000b55b0 0x7f42000ccbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:36.533 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.530+0000 7f4205f52700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42000cbed0 con 0x7f42000cf800 2023-10-06T01:42:36.533 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.530+0000 7f4205f52700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42000cc750 con 0x7f42000d0f00 2023-10-06T01:42:36.533 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.530+0000 7f4205f52700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4200159d40 con 0x7f42000b55b0 2023-10-06T01:42:36.533 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.530+0000 7f41feffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f42000cf800 0x7f42000b5980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47346/0 (socket says 172.21.15.27:47346) 2023-10-06T01:42:36.533 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.530+0000 7f41feffd700 1 -- 172.21.15.27:0/1502754337 learned_addr learned my addr 172.21.15.27:0/1502754337 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:36.533 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.530+0000 7f41ff7fe700 1 --2- 172.21.15.27:0/1502754337 >> v2:172.21.15.124:3300/0 conn(0x7f42000b55b0 0x7f42000ccbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:36.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.530+0000 7f41fe7fc700 1 -- 172.21.15.27:0/1502754337 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2305472344 0 0) 0x7f42000cbed0 con 0x7f42000cf800 2023-10-06T01:42:36.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.530+0000 7f41fe7fc700 1 -- 172.21.15.27:0/1502754337 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41e8003570 con 0x7f42000cf800 2023-10-06T01:42:36.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.530+0000 7f41fe7fc700 1 -- 172.21.15.27:0/1502754337 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4102581782 0 0) 0x7f41e8003570 con 0x7f42000cf800 2023-10-06T01:42:36.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.530+0000 7f41fe7fc700 1 -- 172.21.15.27:0/1502754337 >> v2:172.21.15.124:3300/0 conn(0x7f42000b55b0 msgr2=0x7f42000ccbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:36.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.530+0000 7f41fe7fc700 1 --2- 172.21.15.27:0/1502754337 >> v2:172.21.15.124:3300/0 conn(0x7f42000b55b0 0x7f42000ccbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:36.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.530+0000 7f41fe7fc700 1 -- 172.21.15.27:0/1502754337 >> v1:172.21.15.124:6789/0 conn(0x7f42000d0f00 legacy=0x7f42000ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:36.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.530+0000 7f41fe7fc700 1 -- 172.21.15.27:0/1502754337 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42000cffe0 con 0x7f42000cf800 2023-10-06T01:42:36.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.530+0000 7f41fe7fc700 1 -- 172.21.15.27:0/1502754337 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2619078944 0 0) 0x7f41f4001470 con 0x7f42000cf800 2023-10-06T01:42:36.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.530+0000 7f41fe7fc700 1 -- 172.21.15.27:0/1502754337 >> v1:172.21.15.27:6789/0 conn(0x7f42000cf800 legacy=0x7f42000b5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:36.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.530+0000 7f41fe7fc700 1 --2- 172.21.15.27:0/1502754337 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f41e8003a60 0x7f41e8003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:36.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.530+0000 7f41fe7fc700 1 --2- 172.21.15.27:0/1502754337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41e8004810 0x7f41e8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:36.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.530+0000 7f41fe7fc700 1 --2- 172.21.15.27:0/1502754337 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f41e80052e0 0x7f41e800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:36.534 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.531+0000 7f41fe7fc700 1 -- 172.21.15.27:0/1502754337 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e800c270 con 0x7f41e8004810 2023-10-06T01:42:36.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.531+0000 7f41fe7fc700 1 -- 172.21.15.27:0/1502754337 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41e800c3e0 con 0x7f41e8003a60 2023-10-06T01:42:36.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.531+0000 7f41fe7fc700 1 -- 172.21.15.27:0/1502754337 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41e800c5a0 con 0x7f41e80052e0 2023-10-06T01:42:36.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.531+0000 7f41feffd700 1 --2- 172.21.15.27:0/1502754337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41e8004810 0x7f41e8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:36.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.531+0000 7f41ff7fe700 1 --2- 172.21.15.27:0/1502754337 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f41e8003a60 0x7f41e8003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:36.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.531+0000 7f41fffff700 1 --2- 172.21.15.27:0/1502754337 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f41e80052e0 0x7f41e800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:36.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.531+0000 7f41feffd700 1 -- 172.21.15.27:0/1502754337 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f41e80052e0 msgr2=0x7f41e800ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:36.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.531+0000 7f41feffd700 1 --2- 172.21.15.27:0/1502754337 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f41e80052e0 0x7f41e800ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:36.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.531+0000 7f41feffd700 1 -- 172.21.15.27:0/1502754337 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f41e8003a60 msgr2=0x7f41e8003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:36.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.531+0000 7f41feffd700 1 --2- 172.21.15.27:0/1502754337 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f41e8003a60 0x7f41e8003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:36.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.531+0000 7f41feffd700 1 -- 172.21.15.27:0/1502754337 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e8003570 con 0x7f41e8004810 2023-10-06T01:42:36.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.531+0000 7f41fffff700 1 --2- 172.21.15.27:0/1502754337 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f41e80052e0 0x7f41e800ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:42:36.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.531+0000 7f41fffff700 1 -- 172.21.15.27:0/1502754337 reap_dead start 2023-10-06T01:42:36.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.531+0000 7f41feffd700 1 --2- 172.21.15.27:0/1502754337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41e8004810 0x7f41e8004c80 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f41f400c3c0 tx=0x7f41f4001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1718e6b86b470b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:36.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.531+0000 7f41fe7fc700 1 -- 172.21.15.27:0/1502754337 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f400cc90 con 0x7f41e8004810 2023-10-06T01:42:36.536 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.531+0000 7f41fe7fc700 1 -- 172.21.15.27:0/1502754337 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41f400cdf0 con 0x7f41e8004810 2023-10-06T01:42:36.536 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.531+0000 7f41fe7fc700 1 -- 172.21.15.27:0/1502754337 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f4005b50 con 0x7f41e8004810 2023-10-06T01:42:36.536 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.532+0000 7f4205f52700 1 -- 172.21.15.27:0/1502754337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41e8004810 msgr2=0x7f41e8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:36.536 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.532+0000 7f4205f52700 1 --2- 172.21.15.27:0/1502754337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41e8004810 0x7f41e8004c80 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f41f400c3c0 tx=0x7f41f4001e50 comp rx=0 tx=0).stop 2023-10-06T01:42:36.536 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.532+0000 7f4205f52700 1 -- 172.21.15.27:0/1502754337 shutdown_connections 2023-10-06T01:42:36.536 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.532+0000 7f4205f52700 1 --2- 172.21.15.27:0/1502754337 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f41e8004810 0x7f41e8004c80 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:36.536 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.532+0000 7f4205f52700 1 -- 172.21.15.27:0/1502754337 >> 172.21.15.27:0/1502754337 conn(0x7f420008a050 msgr2=0x7f4200099970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:36.536 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.532+0000 7f4205f52700 1 -- 172.21.15.27:0/1502754337 shutdown_connections 2023-10-06T01:42:36.536 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.532+0000 7f4205f52700 1 -- 172.21.15.27:0/1502754337 wait complete. 2023-10-06T01:42:36.540 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.533+0000 7f4205f52700 1 Processor -- start 2023-10-06T01:42:36.540 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.533+0000 7f4205f52700 1 -- start start 2023-10-06T01:42:36.541 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.534+0000 7f4205f52700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f41e8004810 0x7f420015bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:36.541 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.534+0000 7f4205f52700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f420015ea60 0x7f42000c9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:36.541 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.534+0000 7f4205f52700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f42000ca020 0x7f420015d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:36.541 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.534+0000 7f4205f52700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42000ca490 con 0x7f420015ea60 2023-10-06T01:42:36.541 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.534+0000 7f4205f52700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42000b0740 con 0x7f41e8004810 2023-10-06T01:42:36.541 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.534+0000 7f4205f52700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42000afe50 con 0x7f42000ca020 2023-10-06T01:42:36.541 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.534+0000 7f41feffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f420015ea60 0x7f42000c9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:36.541 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.534+0000 7f41feffd700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f420015ea60 0x7f42000c9ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45814/0 (socket says 172.21.15.27:45814) 2023-10-06T01:42:36.541 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.534+0000 7f41feffd700 1 -- 172.21.15.27:0/929463832 learned_addr learned my addr 172.21.15.27:0/929463832 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:36.541 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.534+0000 7f41ff7fe700 1 --2- 172.21.15.27:0/929463832 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f41e8004810 0x7f420015bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:36.541 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.534+0000 7f41fffff700 1 --2- 172.21.15.27:0/929463832 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f42000ca020 0x7f420015d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:36.542 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.534+0000 7f41feffd700 1 -- 172.21.15.27:0/929463832 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f42000ca020 msgr2=0x7f420015d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:36.542 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.534+0000 7f41feffd700 1 --2- 172.21.15.27:0/929463832 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f42000ca020 0x7f420015d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:36.542 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.534+0000 7f41feffd700 1 -- 172.21.15.27:0/929463832 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f41e8004810 msgr2=0x7f420015bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:36.542 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.534+0000 7f41feffd700 1 --2- 172.21.15.27:0/929463832 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f41e8004810 0x7f420015bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:36.542 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.534+0000 7f41feffd700 1 -- 172.21.15.27:0/929463832 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42000b00f0 con 0x7f420015ea60 2023-10-06T01:42:36.542 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.534+0000 7f41feffd700 1 --2- 172.21.15.27:0/929463832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f420015ea60 0x7f42000c9ae0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f41e800c3b0 tx=0x7f41f4015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:36.542 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.534+0000 7f41e77fe700 1 -- 172.21.15.27:0/929463832 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f4001470 con 0x7f420015ea60 2023-10-06T01:42:36.542 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.535+0000 7f41e77fe700 1 -- 172.21.15.27:0/929463832 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41f40070d0 con 0x7f420015ea60 2023-10-06T01:42:36.542 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.535+0000 7f4205f52700 1 -- 172.21.15.27:0/929463832 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42000b03d0 con 0x7f420015ea60 2023-10-06T01:42:36.542 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.535+0000 7f41e77fe700 1 -- 172.21.15.27:0/929463832 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f4007490 con 0x7f420015ea60 2023-10-06T01:42:36.542 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.535+0000 7f4205f52700 1 -- 172.21.15.27:0/929463832 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f420015c360 con 0x7f420015ea60 2023-10-06T01:42:36.542 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.535+0000 7f4205f52700 1 -- 172.21.15.27:0/929463832 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f42000717c0 con 0x7f420015ea60 2023-10-06T01:42:36.543 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.536+0000 7f41e77fe700 1 -- 172.21.15.27:0/929463832 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f41f4028460 con 0x7f420015ea60 2023-10-06T01:42:36.543 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.536+0000 7f41e77fe700 1 --2- 172.21.15.27:0/929463832 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f41ec066f60 0x7f41ec069410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:36.543 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.536+0000 7f41e77fe700 1 -- 172.21.15.27:0/929463832 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5292+0+0 (secure 0 0 0) 0x7f41f4086400 con 0x7f420015ea60 2023-10-06T01:42:36.543 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.537+0000 7f41e77fe700 1 --2- 172.21.15.27:0/929463832 >> [v2:172.21.15.27:6808/90643,v1:172.21.15.27:6809/90643] conn(0x7f41ec06d3d0 0x7f41ec06f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:36.543 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.537+0000 7f41ff7fe700 1 --2- 172.21.15.27:0/929463832 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f41ec066f60 0x7f41ec069410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:36.543 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.537+0000 7f41e77fe700 1 -- 172.21.15.27:0/929463832 --> [v2:172.21.15.27:6808/90643,v1:172.21.15.27:6809/90643] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f41ec06feb0 con 0x7f41ec06d3d0 2023-10-06T01:42:36.543 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.537+0000 7f41e77fe700 1 -- 172.21.15.27:0/929463832 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f41f40867d0 con 0x7f420015ea60 2023-10-06T01:42:36.543 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.537+0000 7f41fffff700 1 --2- 172.21.15.27:0/929463832 >> [v2:172.21.15.27:6808/90643,v1:172.21.15.27:6809/90643] conn(0x7f41ec06d3d0 0x7f41ec06f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:36.543 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.537+0000 7f41fffff700 1 --2- 172.21.15.27:0/929463832 >> [v2:172.21.15.27:6808/90643,v1:172.21.15.27:6809/90643] conn(0x7f41ec06d3d0 0x7f41ec06f7e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:36.543 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.537+0000 7f41ff7fe700 1 --2- 172.21.15.27:0/929463832 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f41ec066f60 0x7f41ec069410 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f41f0006ea0 tx=0x7f41f0009120 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:36.544 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.538+0000 7f41e77fe700 1 -- 172.21.15.27:0/929463832 <== osd.1 v2:172.21.15.27:6808/90643 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f41ec06feb0 con 0x7f41ec06d3d0 2023-10-06T01:42:36.552 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.551+0000 7f4205f52700 1 -- 172.21.15.27:0/929463832 --> [v2:172.21.15.27:6808/90643,v1:172.21.15.27:6809/90643] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f42000cc100 con 0x7f41ec06d3d0 2023-10-06T01:42:36.552 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.552+0000 7f41e77fe700 1 -- 172.21.15.27:0/929463832 <== osd.1 v2:172.21.15.27:6808/90643 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f42000cc100 con 0x7f41ec06d3d0 2023-10-06T01:42:36.552 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2023-10-06T01:42:36.552 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-06T01:42:36.552 INFO:teuthology.orchestra.run.smithi027.stdout:} 2023-10-06T01:42:36.553 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.552+0000 7f4205f52700 1 -- 172.21.15.27:0/929463832 >> [v2:172.21.15.27:6808/90643,v1:172.21.15.27:6809/90643] conn(0x7f41ec06d3d0 msgr2=0x7f41ec06f7e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:36.553 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.552+0000 7f4205f52700 1 --2- 172.21.15.27:0/929463832 >> [v2:172.21.15.27:6808/90643,v1:172.21.15.27:6809/90643] conn(0x7f41ec06d3d0 0x7f41ec06f7e0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:36.554 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.552+0000 7f4205f52700 1 -- 172.21.15.27:0/929463832 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f41ec066f60 msgr2=0x7f41ec069410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:36.554 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.552+0000 7f4205f52700 1 --2- 172.21.15.27:0/929463832 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f41ec066f60 0x7f41ec069410 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f41f0006ea0 tx=0x7f41f0009120 comp rx=0 tx=0).stop 2023-10-06T01:42:36.554 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.552+0000 7f4205f52700 1 -- 172.21.15.27:0/929463832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f420015ea60 msgr2=0x7f42000c9ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:36.554 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.552+0000 7f4205f52700 1 --2- 172.21.15.27:0/929463832 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f420015ea60 0x7f42000c9ae0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f41e800c3b0 tx=0x7f41f4015000 comp rx=0 tx=0).stop 2023-10-06T01:42:36.554 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.552+0000 7f41fffff700 1 -- 172.21.15.27:0/929463832 reap_dead start 2023-10-06T01:42:36.554 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.552+0000 7f4205f52700 1 -- 172.21.15.27:0/929463832 shutdown_connections 2023-10-06T01:42:36.554 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.552+0000 7f4205f52700 1 -- 172.21.15.27:0/929463832 >> 172.21.15.27:0/929463832 conn(0x7f420008a050 msgr2=0x7f42000ce230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:36.554 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.552+0000 7f4205f52700 1 -- 172.21.15.27:0/929463832 shutdown_connections 2023-10-06T01:42:36.554 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.552+0000 7f4205f52700 1 -- 172.21.15.27:0/929463832 wait complete. 2023-10-06T01:42:36.564 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-06T01:42:36.683 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.683+0000 7ff3adb9c700 1 Processor -- start 2023-10-06T01:42:36.684 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.683+0000 7ff3adb9c700 1 -- start start 2023-10-06T01:42:36.685 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.684+0000 7ff3adb9c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff3a8068350 0x7ff3a80687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:36.686 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.684+0000 7ff3adb9c700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a8068cf0 con 0x7ff3a81593f0 2023-10-06T01:42:36.686 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.684+0000 7ff3adb9c700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a80cab50 con 0x7ff3a81597c0 2023-10-06T01:42:36.686 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.684+0000 7ff3adb9c700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3a80cb3f0 con 0x7ff3a8068350 2023-10-06T01:42:36.686 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.684+0000 7ff3ad39b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7ff3a8068350 0x7ff3a80687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:36.686 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.684+0000 7ff3acb9a700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7ff3a81593f0 0x7ff3a80b00b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47354/0 (socket says 172.21.15.27:47354) 2023-10-06T01:42:36.686 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.684+0000 7ff3acb9a700 1 -- 172.21.15.27:0/1702043068 learned_addr learned my addr 172.21.15.27:0/1702043068 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:36.686 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.684+0000 7ff3ad39b700 1 -- 172.21.15.27:0/1702043068 >> v1:172.21.15.124:6789/0 conn(0x7ff3a81597c0 legacy=0x7ff3a80cf680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:36.687 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.684+0000 7ff3ad39b700 1 -- 172.21.15.27:0/1702043068 >> v1:172.21.15.27:6789/0 conn(0x7ff3a81593f0 legacy=0x7ff3a80b00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:36.687 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.684+0000 7ff3ad39b700 1 -- 172.21.15.27:0/1702043068 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a80d0150 con 0x7ff3a8068350 2023-10-06T01:42:36.687 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.685+0000 7ff3ad39b700 1 --2- 172.21.15.27:0/1702043068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3a8068350 0x7ff3a80687b0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff3980098e0 tx=0x7ff3980091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6d7cfc2c9b6a6b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:36.687 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.685+0000 7ff3a77fe700 1 -- 172.21.15.27:0/1702043068 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff398009e80 con 0x7ff3a8068350 2023-10-06T01:42:36.687 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.685+0000 7ff3a77fe700 1 -- 172.21.15.27:0/1702043068 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff39800b860 con 0x7ff3a8068350 2023-10-06T01:42:36.687 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.685+0000 7ff3a77fe700 1 -- 172.21.15.27:0/1702043068 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff39800bbe0 con 0x7ff3a8068350 2023-10-06T01:42:36.687 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.685+0000 7ff3adb9c700 1 -- 172.21.15.27:0/1702043068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3a8068350 msgr2=0x7ff3a80687b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:36.687 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.685+0000 7ff3adb9c700 1 --2- 172.21.15.27:0/1702043068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3a8068350 0x7ff3a80687b0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7ff3980098e0 tx=0x7ff3980091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:36.687 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.685+0000 7ff3adb9c700 1 -- 172.21.15.27:0/1702043068 shutdown_connections 2023-10-06T01:42:36.688 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.685+0000 7ff3adb9c700 1 --2- 172.21.15.27:0/1702043068 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3a8068350 0x7ff3a80687b0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:36.688 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.685+0000 7ff3adb9c700 1 -- 172.21.15.27:0/1702043068 >> 172.21.15.27:0/1702043068 conn(0x7ff3a80d0510 msgr2=0x7ff3a80d0910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:36.688 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.685+0000 7ff3adb9c700 1 -- 172.21.15.27:0/1702043068 shutdown_connections 2023-10-06T01:42:36.688 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.685+0000 7ff3adb9c700 1 -- 172.21.15.27:0/1702043068 wait complete. 2023-10-06T01:42:36.691 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.686+0000 7ff3adb9c700 1 Processor -- start 2023-10-06T01:42:36.691 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.687+0000 7ff3adb9c700 1 -- start start 2023-10-06T01:42:36.691 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.687+0000 7ff3adb9c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3a8068350 0x7ff3a8153e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:36.691 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.687+0000 7ff3adb9c700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff3a81593f0 0x7ff3a815e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:36.691 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.687+0000 7ff3adb9c700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff3a81597c0 0x7ff3a8158060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:36.691 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.687+0000 7ff3adb9c700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a8157bd0 con 0x7ff3a8068350 2023-10-06T01:42:36.691 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.687+0000 7ff3adb9c700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a815c650 con 0x7ff3a81593f0 2023-10-06T01:42:36.692 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.687+0000 7ff3adb9c700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a815c930 con 0x7ff3a81597c0 2023-10-06T01:42:36.692 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.687+0000 7ff3acb9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3a8068350 0x7ff3a8153e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:36.692 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.687+0000 7ff3acb9a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3a8068350 0x7ff3a8153e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45834/0 (socket says 172.21.15.27:45834) 2023-10-06T01:42:36.692 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.687+0000 7ff3acb9a700 1 -- 172.21.15.27:0/3643401801 learned_addr learned my addr 172.21.15.27:0/3643401801 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:36.692 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.688+0000 7ff3ad39b700 1 --2- 172.21.15.27:0/3643401801 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff3a81597c0 0x7ff3a8158060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:36.692 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.688+0000 7ff3a7fff700 1 --2- 172.21.15.27:0/3643401801 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff3a81593f0 0x7ff3a815e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:36.692 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.688+0000 7ff3acb9a700 1 -- 172.21.15.27:0/3643401801 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff3a81597c0 msgr2=0x7ff3a8158060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:36.692 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.688+0000 7ff3acb9a700 1 --2- 172.21.15.27:0/3643401801 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff3a81597c0 0x7ff3a8158060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:36.692 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.688+0000 7ff3acb9a700 1 -- 172.21.15.27:0/3643401801 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff3a81593f0 msgr2=0x7ff3a815e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:36.693 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.688+0000 7ff3acb9a700 1 --2- 172.21.15.27:0/3643401801 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff3a81593f0 0x7ff3a815e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:36.693 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.688+0000 7ff3acb9a700 1 -- 172.21.15.27:0/3643401801 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff398009420 con 0x7ff3a8068350 2023-10-06T01:42:36.693 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.688+0000 7ff3ad39b700 1 --2- 172.21.15.27:0/3643401801 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff3a81597c0 0x7ff3a8158060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:42:36.693 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.688+0000 7ff3a7fff700 1 --2- 172.21.15.27:0/3643401801 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff3a81593f0 0x7ff3a815e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:42:36.693 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.688+0000 7ff3acb9a700 1 --2- 172.21.15.27:0/3643401801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3a8068350 0x7ff3a8153e40 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff3a000cd30 tx=0x7ff3a0001ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:36.693 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.688+0000 7ff3a4ff9700 1 -- 172.21.15.27:0/3643401801 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a0004bb0 con 0x7ff3a8068350 2023-10-06T01:42:36.693 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.688+0000 7ff3a4ff9700 1 -- 172.21.15.27:0/3643401801 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3a0004d10 con 0x7ff3a8068350 2023-10-06T01:42:36.693 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.688+0000 7ff3adb9c700 1 -- 172.21.15.27:0/3643401801 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3a815cc10 con 0x7ff3a8068350 2023-10-06T01:42:36.693 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.688+0000 7ff3a4ff9700 1 -- 172.21.15.27:0/3643401801 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a000adc0 con 0x7ff3a8068350 2023-10-06T01:42:36.694 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.688+0000 7ff3adb9c700 1 -- 172.21.15.27:0/3643401801 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3a815ecf0 con 0x7ff3a8068350 2023-10-06T01:42:36.694 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.689+0000 7ff3adb9c700 1 -- 172.21.15.27:0/3643401801 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3a804c3c0 con 0x7ff3a8068350 2023-10-06T01:42:36.694 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.689+0000 7ff3a4ff9700 1 -- 172.21.15.27:0/3643401801 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7ff3a0007300 con 0x7ff3a8068350 2023-10-06T01:42:36.694 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.690+0000 7ff3a4ff9700 1 --2- 172.21.15.27:0/3643401801 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7ff394066f30 0x7ff3940693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:36.694 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.690+0000 7ff3a4ff9700 1 -- 172.21.15.27:0/3643401801 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5292+0+0 (secure 0 0 0) 0x7ff3a0083f70 con 0x7ff3a8068350 2023-10-06T01:42:36.694 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.690+0000 7ff3a7fff700 1 --2- 172.21.15.27:0/3643401801 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7ff394066f30 0x7ff3940693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:36.694 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.691+0000 7ff3a7fff700 1 --2- 172.21.15.27:0/3643401801 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7ff394066f30 0x7ff3940693e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff39c006fd0 tx=0x7ff39c008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:36.694 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.694+0000 7ff3a4ff9700 1 -- 172.21.15.27:0/3643401801 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff3a0051400 con 0x7ff3a8068350 2023-10-06T01:42:36.886 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.886+0000 7ff3adb9c700 1 -- 172.21.15.27:0/3643401801 --> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7ff3a8157d10 con 0x7ff394066f30 2023-10-06T01:42:36.886 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.886+0000 7ff3a4ff9700 1 -- 172.21.15.27:0/3643401801 <== mgr.4100 v2:172.21.15.124:6832/90278 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff39c00ad80 con 0x7ff394066f30 2023-10-06T01:42:36.889 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.888+0000 7ff3adb9c700 1 -- 172.21.15.27:0/3643401801 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7ff394066f30 msgr2=0x7ff3940693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:36.889 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.888+0000 7ff3adb9c700 1 --2- 172.21.15.27:0/3643401801 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7ff394066f30 0x7ff3940693e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff39c006fd0 tx=0x7ff39c008040 comp rx=0 tx=0).stop 2023-10-06T01:42:36.889 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.888+0000 7ff3adb9c700 1 -- 172.21.15.27:0/3643401801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3a8068350 msgr2=0x7ff3a8153e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:36.889 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.888+0000 7ff3adb9c700 1 --2- 172.21.15.27:0/3643401801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3a8068350 0x7ff3a8153e40 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7ff3a000cd30 tx=0x7ff3a0001ad0 comp rx=0 tx=0).stop 2023-10-06T01:42:36.889 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.888+0000 7ff3adb9c700 1 -- 172.21.15.27:0/3643401801 shutdown_connections 2023-10-06T01:42:36.889 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.888+0000 7ff3adb9c700 1 --2- 172.21.15.27:0/3643401801 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7ff394066f30 0x7ff3940693e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:36.889 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.888+0000 7ff3adb9c700 1 --2- 172.21.15.27:0/3643401801 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7ff3a81597c0 0x7ff3a8158060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:36.889 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.888+0000 7ff3adb9c700 1 --2- 172.21.15.27:0/3643401801 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7ff3a8068350 0x7ff3a8153e40 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:36.890 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.888+0000 7ff3adb9c700 1 --2- 172.21.15.27:0/3643401801 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7ff3a81593f0 0x7ff3a815e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:36.890 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.888+0000 7ff3adb9c700 1 -- 172.21.15.27:0/3643401801 >> 172.21.15.27:0/3643401801 conn(0x7ff3a80d0510 msgr2=0x7ff3a80c52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:36.890 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.888+0000 7ff3adb9c700 1 -- 172.21.15.27:0/3643401801 shutdown_connections 2023-10-06T01:42:36.890 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:36.888+0000 7ff3adb9c700 1 -- 172.21.15.27:0/3643401801 wait complete. 2023-10-06T01:42:36.890 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 1 to deep-scrub 2023-10-06T01:42:36.900 INFO:tasks.ceph:Scrubbing osd.2 2023-10-06T01:42:36.901 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-06T01:42:37.023 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.023+0000 7f1131d9d700 1 Processor -- start 2023-10-06T01:42:37.023 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.023+0000 7f1131d9d700 1 -- start start 2023-10-06T01:42:37.027 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.024+0000 7f1131d9d700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f112c0b55b0 0x7f112c0ccbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:37.027 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.024+0000 7f1131d9d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f112c0cf800 0x7f112c0b5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:37.027 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.024+0000 7f1131d9d700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f112c0cbed0 con 0x7f112c0d0f00 2023-10-06T01:42:37.027 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.024+0000 7f1131d9d700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f112c159d40 con 0x7f112c0b55b0 2023-10-06T01:42:37.027 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.024+0000 7f1131d9d700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f112c0cc750 con 0x7f112c0cf800 2023-10-06T01:42:37.028 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.024+0000 7f112bfff700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f112c0d0f00 0x7f112c0ce270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47358/0 (socket says 172.21.15.27:47358) 2023-10-06T01:42:37.028 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.024+0000 7f112bfff700 1 -- 172.21.15.27:0/2371238773 learned_addr learned my addr 172.21.15.27:0/2371238773 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:37.028 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.024+0000 7f112b7fe700 1 --2- 172.21.15.27:0/2371238773 >> v2:172.21.15.124:3300/0 conn(0x7f112c0b55b0 0x7f112c0ccbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:37.028 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.024+0000 7f112affd700 1 --2- 172.21.15.27:0/2371238773 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f112c0cf800 0x7f112c0b5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:37.028 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.024+0000 7f112a7fc700 1 -- 172.21.15.27:0/2371238773 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 611583744 0 0) 0x7f112c0cbed0 con 0x7f112c0d0f00 2023-10-06T01:42:37.028 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.024+0000 7f112a7fc700 1 -- 172.21.15.27:0/2371238773 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1114003570 con 0x7f112c0d0f00 2023-10-06T01:42:37.028 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.024+0000 7f112a7fc700 1 -- 172.21.15.27:0/2371238773 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1779681604 0 0) 0x7f1114003570 con 0x7f112c0d0f00 2023-10-06T01:42:37.028 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.025+0000 7f112a7fc700 1 -- 172.21.15.27:0/2371238773 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f112c0cf800 msgr2=0x7f112c0b5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:37.028 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.025+0000 7f112a7fc700 1 --2- 172.21.15.27:0/2371238773 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f112c0cf800 0x7f112c0b5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:37.028 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.025+0000 7f112a7fc700 1 -- 172.21.15.27:0/2371238773 >> v2:172.21.15.124:3300/0 conn(0x7f112c0b55b0 msgr2=0x7f112c0ccbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:37.028 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.025+0000 7f112a7fc700 1 --2- 172.21.15.27:0/2371238773 >> v2:172.21.15.124:3300/0 conn(0x7f112c0b55b0 0x7f112c0ccbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:37.028 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.025+0000 7f112a7fc700 1 -- 172.21.15.27:0/2371238773 --> v1:172.21.15.27:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f112c0b5ec0 con 0x7f112c0d0f00 2023-10-06T01:42:37.029 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.025+0000 7f112a7fc700 1 -- 172.21.15.27:0/2371238773 <== mon.0 v1:172.21.15.27:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2619078944 0 0) 0x7f11240017f0 con 0x7f112c0d0f00 2023-10-06T01:42:37.029 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.025+0000 7f112a7fc700 1 -- 172.21.15.27:0/2371238773 >> v1:172.21.15.27:6789/0 conn(0x7f112c0d0f00 legacy=0x7f112c0ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:37.029 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.025+0000 7f112a7fc700 1 --2- 172.21.15.27:0/2371238773 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1114003a60 0x7f1114003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:37.029 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.025+0000 7f112a7fc700 1 --2- 172.21.15.27:0/2371238773 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f112c0cf800 0x7f1114004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:37.029 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.025+0000 7f112a7fc700 1 --2- 172.21.15.27:0/2371238773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1114004d50 0x7f11140051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:37.029 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.025+0000 7f112a7fc700 1 -- 172.21.15.27:0/2371238773 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f112c0cc750 con 0x7f1114004d50 2023-10-06T01:42:37.029 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.025+0000 7f112a7fc700 1 -- 172.21.15.27:0/2371238773 --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f111400ba60 con 0x7f1114003a60 2023-10-06T01:42:37.029 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.025+0000 7f112a7fc700 1 -- 172.21.15.27:0/2371238773 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f111400bc20 con 0x7f112c0cf800 2023-10-06T01:42:37.029 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.025+0000 7f112b7fe700 1 --2- 172.21.15.27:0/2371238773 >> v2:172.21.15.124:3300/0 conn(0x7f112c0b55b0 0x7f112c0ccbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:42:37.030 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.025+0000 7f112bfff700 1 --2- 172.21.15.27:0/2371238773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1114004d50 0x7f11140051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:37.030 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.025+0000 7f112affd700 1 --2- 172.21.15.27:0/2371238773 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f112c0cf800 0x7f1114004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:37.030 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.025+0000 7f112b7fe700 1 --2- 172.21.15.27:0/2371238773 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1114003a60 0x7f1114003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:37.030 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.025+0000 7f112bfff700 1 -- 172.21.15.27:0/2371238773 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f112c0cf800 msgr2=0x7f1114004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:37.030 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.025+0000 7f112bfff700 1 --2- 172.21.15.27:0/2371238773 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f112c0cf800 0x7f1114004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:37.030 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.025+0000 7f112bfff700 1 -- 172.21.15.27:0/2371238773 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1114003a60 msgr2=0x7f1114003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:37.030 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.025+0000 7f112bfff700 1 --2- 172.21.15.27:0/2371238773 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1114003a60 0x7f1114003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:37.030 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.025+0000 7f112bfff700 1 -- 172.21.15.27:0/2371238773 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1114003570 con 0x7f1114004d50 2023-10-06T01:42:37.030 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.025+0000 7f112b7fe700 1 --2- 172.21.15.27:0/2371238773 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f1114003a60 0x7f1114003e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:42:37.030 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.025+0000 7f112bfff700 1 --2- 172.21.15.27:0/2371238773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1114004d50 0x7f11140051c0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f112400a970 tx=0x7f1124002210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cde8cc73a5ed9796 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:37.030 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.025+0000 7f112a7fc700 1 -- 172.21.15.27:0/2371238773 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1124012440 con 0x7f1114004d50 2023-10-06T01:42:37.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.025+0000 7f112a7fc700 1 -- 172.21.15.27:0/2371238773 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11240125c0 con 0x7f1114004d50 2023-10-06T01:42:37.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.025+0000 7f112a7fc700 1 -- 172.21.15.27:0/2371238773 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1124012980 con 0x7f1114004d50 2023-10-06T01:42:37.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.026+0000 7f1131d9d700 1 -- 172.21.15.27:0/2371238773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1114004d50 msgr2=0x7f11140051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:37.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.026+0000 7f1131d9d700 1 --2- 172.21.15.27:0/2371238773 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1114004d50 0x7f11140051c0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f112400a970 tx=0x7f1124002210 comp rx=0 tx=0).stop 2023-10-06T01:42:37.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.026+0000 7f112bfff700 1 -- 172.21.15.27:0/2371238773 reap_dead start 2023-10-06T01:42:37.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.026+0000 7f1131d9d700 1 -- 172.21.15.27:0/2371238773 shutdown_connections 2023-10-06T01:42:37.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.026+0000 7f1131d9d700 1 -- 172.21.15.27:0/2371238773 >> 172.21.15.27:0/2371238773 conn(0x7f112c08a050 msgr2=0x7f112c099970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:37.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.026+0000 7f1131d9d700 1 -- 172.21.15.27:0/2371238773 shutdown_connections 2023-10-06T01:42:37.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.026+0000 7f1131d9d700 1 -- 172.21.15.27:0/2371238773 wait complete. 2023-10-06T01:42:37.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.027+0000 7f1131d9d700 1 Processor -- start 2023-10-06T01:42:37.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.027+0000 7f1131d9d700 1 -- start start 2023-10-06T01:42:37.034 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.028+0000 7f1131d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1114004d50 0x7f112c15c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:37.034 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.028+0000 7f1131d9d700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f112c158700 0x7f112c0b07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:37.034 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.028+0000 7f1131d9d700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f112c15ab70 0x7f112c15afe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:37.034 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.028+0000 7f1131d9d700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f112c0b18c0 con 0x7f1114004d50 2023-10-06T01:42:37.034 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.028+0000 7f1131d9d700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f112c15b520 con 0x7f112c158700 2023-10-06T01:42:37.034 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.028+0000 7f1131d9d700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f112c0b0d00 con 0x7f112c15ab70 2023-10-06T01:42:37.035 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.028+0000 7f112b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1114004d50 0x7f112c15c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:37.035 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.028+0000 7f112b7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1114004d50 0x7f112c15c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45852/0 (socket says 172.21.15.27:45852) 2023-10-06T01:42:37.035 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.028+0000 7f112b7fe700 1 -- 172.21.15.27:0/3736493622 learned_addr learned my addr 172.21.15.27:0/3736493622 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:37.035 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.028+0000 7f112affd700 1 --2- 172.21.15.27:0/3736493622 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f112c158700 0x7f112c0b07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:37.035 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.028+0000 7f112bfff700 1 --2- 172.21.15.27:0/3736493622 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f112c15ab70 0x7f112c15afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:37.035 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.028+0000 7f112b7fe700 1 -- 172.21.15.27:0/3736493622 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f112c15ab70 msgr2=0x7f112c15afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:37.035 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.028+0000 7f112b7fe700 1 --2- 172.21.15.27:0/3736493622 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f112c15ab70 0x7f112c15afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:37.035 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.028+0000 7f112b7fe700 1 -- 172.21.15.27:0/3736493622 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f112c158700 msgr2=0x7f112c0b07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:37.035 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.028+0000 7f112b7fe700 1 --2- 172.21.15.27:0/3736493622 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f112c158700 0x7f112c0b07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:37.035 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.028+0000 7f112b7fe700 1 -- 172.21.15.27:0/3736493622 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f112c15ca90 con 0x7f1114004d50 2023-10-06T01:42:37.035 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.029+0000 7f112b7fe700 1 --2- 172.21.15.27:0/3736493622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1114004d50 0x7f112c15c4b0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f111c011970 tx=0x7f111c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:37.035 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.029+0000 7f11137fe700 1 -- 172.21.15.27:0/3736493622 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f111c00f8e0 con 0x7f1114004d50 2023-10-06T01:42:37.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.029+0000 7f11137fe700 1 -- 172.21.15.27:0/3736493622 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f111c00fa40 con 0x7f1114004d50 2023-10-06T01:42:37.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.029+0000 7f1131d9d700 1 -- 172.21.15.27:0/3736493622 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f112c15cd70 con 0x7f1114004d50 2023-10-06T01:42:37.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.029+0000 7f11137fe700 1 -- 172.21.15.27:0/3736493622 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f111c00fdc0 con 0x7f1114004d50 2023-10-06T01:42:37.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.029+0000 7f1131d9d700 1 -- 172.21.15.27:0/3736493622 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f112c15d330 con 0x7f1114004d50 2023-10-06T01:42:37.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.029+0000 7f1131d9d700 1 -- 172.21.15.27:0/3736493622 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f112c0cf3c0 con 0x7f1114004d50 2023-10-06T01:42:37.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.030+0000 7f11137fe700 1 -- 172.21.15.27:0/3736493622 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f111c015670 con 0x7f1114004d50 2023-10-06T01:42:37.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.030+0000 7f11137fe700 1 --2- 172.21.15.27:0/3736493622 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1118066f30 0x7f11180693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:37.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.031+0000 7f11137fe700 1 -- 172.21.15.27:0/3736493622 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5292+0+0 (secure 0 0 0) 0x7f111c084d80 con 0x7f1114004d50 2023-10-06T01:42:37.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.031+0000 7f112affd700 1 --2- 172.21.15.27:0/3736493622 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1118066f30 0x7f11180693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:37.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.031+0000 7f11137fe700 1 --2- 172.21.15.27:0/3736493622 >> [v2:172.21.15.27:6816/90645,v1:172.21.15.27:6817/90645] conn(0x7f111806d3a0 0x7f111806f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:37.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.031+0000 7f11137fe700 1 -- 172.21.15.27:0/3736493622 --> [v2:172.21.15.27:6816/90645,v1:172.21.15.27:6817/90645] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f111806fe80 con 0x7f111806d3a0 2023-10-06T01:42:37.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.031+0000 7f11137fe700 1 -- 172.21.15.27:0/3736493622 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f111c087360 con 0x7f1114004d50 2023-10-06T01:42:37.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.031+0000 7f112bfff700 1 --2- 172.21.15.27:0/3736493622 >> [v2:172.21.15.27:6816/90645,v1:172.21.15.27:6817/90645] conn(0x7f111806d3a0 0x7f111806f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:37.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.031+0000 7f112affd700 1 --2- 172.21.15.27:0/3736493622 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1118066f30 0x7f11180693e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f112c0af860 tx=0x7f1120008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:37.036 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.031+0000 7f112bfff700 1 --2- 172.21.15.27:0/3736493622 >> [v2:172.21.15.27:6816/90645,v1:172.21.15.27:6817/90645] conn(0x7f111806d3a0 0x7f111806f7b0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:37.037 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.032+0000 7f11137fe700 1 -- 172.21.15.27:0/3736493622 <== osd.2 v2:172.21.15.27:6816/90645 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f111806fe80 con 0x7f111806d3a0 2023-10-06T01:42:37.046 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.045+0000 7f1131d9d700 1 -- 172.21.15.27:0/3736493622 --> [v2:172.21.15.27:6816/90645,v1:172.21.15.27:6817/90645] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f112c0717c0 con 0x7f111806d3a0 2023-10-06T01:42:37.046 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.046+0000 7f11137fe700 1 -- 172.21.15.27:0/3736493622 <== osd.2 v2:172.21.15.27:6816/90645 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f112c0717c0 con 0x7f111806d3a0 2023-10-06T01:42:37.046 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2023-10-06T01:42:37.046 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-06T01:42:37.046 INFO:teuthology.orchestra.run.smithi027.stdout:} 2023-10-06T01:42:37.047 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.046+0000 7f1131d9d700 1 -- 172.21.15.27:0/3736493622 >> [v2:172.21.15.27:6816/90645,v1:172.21.15.27:6817/90645] conn(0x7f111806d3a0 msgr2=0x7f111806f7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:37.047 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.046+0000 7f1131d9d700 1 --2- 172.21.15.27:0/3736493622 >> [v2:172.21.15.27:6816/90645,v1:172.21.15.27:6817/90645] conn(0x7f111806d3a0 0x7f111806f7b0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:37.047 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.046+0000 7f1131d9d700 1 -- 172.21.15.27:0/3736493622 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1118066f30 msgr2=0x7f11180693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:37.047 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.046+0000 7f1131d9d700 1 --2- 172.21.15.27:0/3736493622 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f1118066f30 0x7f11180693e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f112c0af860 tx=0x7f1120008040 comp rx=0 tx=0).stop 2023-10-06T01:42:37.048 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.046+0000 7f1131d9d700 1 -- 172.21.15.27:0/3736493622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1114004d50 msgr2=0x7f112c15c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:37.048 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.046+0000 7f1131d9d700 1 --2- 172.21.15.27:0/3736493622 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f1114004d50 0x7f112c15c4b0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f111c011970 tx=0x7f111c008040 comp rx=0 tx=0).stop 2023-10-06T01:42:37.048 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.046+0000 7f112bfff700 1 -- 172.21.15.27:0/3736493622 reap_dead start 2023-10-06T01:42:37.048 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.046+0000 7f1131d9d700 1 -- 172.21.15.27:0/3736493622 shutdown_connections 2023-10-06T01:42:37.048 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.046+0000 7f1131d9d700 1 -- 172.21.15.27:0/3736493622 >> 172.21.15.27:0/3736493622 conn(0x7f112c08a050 msgr2=0x7f112c159b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:37.048 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.046+0000 7f1131d9d700 1 -- 172.21.15.27:0/3736493622 shutdown_connections 2023-10-06T01:42:37.048 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.046+0000 7f1131d9d700 1 -- 172.21.15.27:0/3736493622 wait complete. 2023-10-06T01:42:37.058 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-06T01:42:37.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.177+0000 7f6b6b59e700 1 Processor -- start 2023-10-06T01:42:37.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.177+0000 7f6b6b59e700 1 -- start start 2023-10-06T01:42:37.179 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.178+0000 7f6b6b59e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6b6c159910 0x7f6b6c0c8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:37.179 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.178+0000 7f6b6b59e700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6b6c0c9510 0x7f6b6c0aee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:37.179 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.178+0000 7f6b6b59e700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b6c0c7bf0 con 0x7f6b6c159540 2023-10-06T01:42:37.179 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.178+0000 7f6b6b59e700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b6c0c9990 con 0x7f6b6c0c9510 2023-10-06T01:42:37.179 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.178+0000 7f6b6b59e700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6c0c8a50 con 0x7f6b6c159910 2023-10-06T01:42:37.179 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.178+0000 7f6b6ad9d700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6b6c0c9510 0x7f6b6c0aee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:37.180 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.178+0000 7f6b6a59c700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f6b6c159540 0x7f6b6c15dc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47368/0 (socket says 172.21.15.27:47368) 2023-10-06T01:42:37.180 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.178+0000 7f6b6a59c700 1 -- 172.21.15.27:0/2639318682 learned_addr learned my addr 172.21.15.27:0/2639318682 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:37.180 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.178+0000 7f6b69d9b700 1 --2- 172.21.15.27:0/2639318682 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6b6c159910 0x7f6b6c0c8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:37.180 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.178+0000 7f6b6ad9d700 1 -- 172.21.15.27:0/2639318682 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6b6c159910 msgr2=0x7f6b6c0c8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:37.180 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.178+0000 7f6b6ad9d700 1 --2- 172.21.15.27:0/2639318682 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6b6c159910 0x7f6b6c0c8fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:37.180 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.178+0000 7f6b6ad9d700 1 -- 172.21.15.27:0/2639318682 >> v1:172.21.15.27:6789/0 conn(0x7f6b6c159540 legacy=0x7f6b6c15dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:37.180 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.178+0000 7f6b6959a700 1 -- 172.21.15.27:0/2639318682 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3863297762 0 0) 0x7f6b6c0c7bf0 con 0x7f6b6c159540 2023-10-06T01:42:37.180 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.178+0000 7f6b6ad9d700 1 -- 172.21.15.27:0/2639318682 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b6c0c7e40 con 0x7f6b6c0c9510 2023-10-06T01:42:37.180 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.179+0000 7f6b6ad9d700 1 --2- 172.21.15.27:0/2639318682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b6c0c9510 0x7f6b6c0aee20 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c009e60 tx=0x7f6b5c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2d4b16fc84c8d1 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:37.180 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.179+0000 7f6b6959a700 1 -- 172.21.15.27:0/2639318682 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b5c00e070 con 0x7f6b6c0c9510 2023-10-06T01:42:37.180 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.179+0000 7f6b6959a700 1 -- 172.21.15.27:0/2639318682 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b5c00bb60 con 0x7f6b6c0c9510 2023-10-06T01:42:37.180 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.179+0000 7f6b6959a700 1 -- 172.21.15.27:0/2639318682 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b5c012430 con 0x7f6b6c0c9510 2023-10-06T01:42:37.183 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.179+0000 7f6b6b59e700 1 -- 172.21.15.27:0/2639318682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b6c0c9510 msgr2=0x7f6b6c0aee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:37.183 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.179+0000 7f6b6b59e700 1 --2- 172.21.15.27:0/2639318682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b6c0c9510 0x7f6b6c0aee20 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f6b5c009e60 tx=0x7f6b5c009780 comp rx=0 tx=0).stop 2023-10-06T01:42:37.184 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.179+0000 7f6b6b59e700 1 -- 172.21.15.27:0/2639318682 shutdown_connections 2023-10-06T01:42:37.184 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.179+0000 7f6b6b59e700 1 --2- 172.21.15.27:0/2639318682 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b6c0c9510 0x7f6b6c0aee20 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:37.184 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.179+0000 7f6b6b59e700 1 --2- 172.21.15.27:0/2639318682 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6b6c159910 0x7f6b6c0c8fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:37.184 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.179+0000 7f6b6b59e700 1 -- 172.21.15.27:0/2639318682 >> 172.21.15.27:0/2639318682 conn(0x7f6b6c06e600 msgr2=0x7f6b6c06ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:37.184 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.179+0000 7f6b6b59e700 1 -- 172.21.15.27:0/2639318682 shutdown_connections 2023-10-06T01:42:37.184 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.180+0000 7f6b6b59e700 1 -- 172.21.15.27:0/2639318682 wait complete. 2023-10-06T01:42:37.184 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.181+0000 7f6b6b59e700 1 Processor -- start 2023-10-06T01:42:37.184 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.181+0000 7f6b6b59e700 1 -- start start 2023-10-06T01:42:37.184 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.181+0000 7f6b6b59e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6b6c0c9510 0x7f6b6c16cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:37.184 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.181+0000 7f6b6b59e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b6c159540 0x7f6b6c16d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:37.185 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.181+0000 7f6b6b59e700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6b6c159910 0x7f6b6c144420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:37.185 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.181+0000 7f6b6b59e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6c1600f0 con 0x7f6b6c159540 2023-10-06T01:42:37.185 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.181+0000 7f6b6b59e700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6c0c5ca0 con 0x7f6b6c159910 2023-10-06T01:42:37.185 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.181+0000 7f6b6b59e700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b6c152490 con 0x7f6b6c0c9510 2023-10-06T01:42:37.185 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.181+0000 7f6b69d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b6c159540 0x7f6b6c16d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:37.185 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.181+0000 7f6b69d9b700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b6c159540 0x7f6b6c16d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45870/0 (socket says 172.21.15.27:45870) 2023-10-06T01:42:37.185 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.181+0000 7f6b69d9b700 1 -- 172.21.15.27:0/1094155666 learned_addr learned my addr 172.21.15.27:0/1094155666 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:37.185 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.181+0000 7f6b6a59c700 1 --2- 172.21.15.27:0/1094155666 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6b6c0c9510 0x7f6b6c16cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:37.185 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.182+0000 7f6b6ad9d700 1 --2- 172.21.15.27:0/1094155666 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6b6c159910 0x7f6b6c144420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:37.185 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.182+0000 7f6b69d9b700 1 -- 172.21.15.27:0/1094155666 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6b6c0c9510 msgr2=0x7f6b6c16cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:37.185 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.182+0000 7f6b69d9b700 1 --2- 172.21.15.27:0/1094155666 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6b6c0c9510 0x7f6b6c16cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:37.185 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.182+0000 7f6b69d9b700 1 -- 172.21.15.27:0/1094155666 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6b6c159910 msgr2=0x7f6b6c144420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:37.185 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.182+0000 7f6b69d9b700 1 --2- 172.21.15.27:0/1094155666 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6b6c159910 0x7f6b6c144420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:37.186 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.182+0000 7f6b69d9b700 1 -- 172.21.15.27:0/1094155666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b5c009a20 con 0x7f6b6c159540 2023-10-06T01:42:37.186 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.182+0000 7f6b69d9b700 1 --2- 172.21.15.27:0/1094155666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b6c159540 0x7f6b6c16d3e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6b5400e900 tx=0x7f6b540091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:37.186 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.182+0000 7f6b627fc700 1 -- 172.21.15.27:0/1094155666 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b54013680 con 0x7f6b6c159540 2023-10-06T01:42:37.186 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.182+0000 7f6b627fc700 1 -- 172.21.15.27:0/1094155666 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b540137e0 con 0x7f6b6c159540 2023-10-06T01:42:37.186 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.182+0000 7f6b6b59e700 1 -- 172.21.15.27:0/1094155666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b6c0c64e0 con 0x7f6b6c159540 2023-10-06T01:42:37.186 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.182+0000 7f6b627fc700 1 -- 172.21.15.27:0/1094155666 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b54013ba0 con 0x7f6b6c159540 2023-10-06T01:42:37.186 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.182+0000 7f6b6b59e700 1 -- 172.21.15.27:0/1094155666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b6c0c6170 con 0x7f6b6c159540 2023-10-06T01:42:37.186 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.183+0000 7f6b6b59e700 1 -- 172.21.15.27:0/1094155666 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b6c0c6670 con 0x7f6b6c159540 2023-10-06T01:42:37.186 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.183+0000 7f6b627fc700 1 -- 172.21.15.27:0/1094155666 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6b540123d0 con 0x7f6b6c159540 2023-10-06T01:42:37.187 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.184+0000 7f6b627fc700 1 --2- 172.21.15.27:0/1094155666 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6b4c066f30 0x7f6b4c0693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:37.187 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.184+0000 7f6b627fc700 1 -- 172.21.15.27:0/1094155666 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5292+0+0 (secure 0 0 0) 0x7f6b54084440 con 0x7f6b6c159540 2023-10-06T01:42:37.187 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.184+0000 7f6b6a59c700 1 --2- 172.21.15.27:0/1094155666 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6b4c066f30 0x7f6b4c0693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:37.187 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.184+0000 7f6b6a59c700 1 --2- 172.21.15.27:0/1094155666 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6b4c066f30 0x7f6b4c0693e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6b64007fd0 tx=0x7f6b64007d30 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:37.188 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.188+0000 7f6b627fc700 1 -- 172.21.15.27:0/1094155666 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6b54051850 con 0x7f6b6c159540 2023-10-06T01:42:37.382 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.381+0000 7f6b6b59e700 1 -- 172.21.15.27:0/1094155666 --> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b6c04c3c0 con 0x7f6b4c066f30 2023-10-06T01:42:37.382 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.382+0000 7f6b627fc700 1 -- 172.21.15.27:0/1094155666 <== mgr.4100 v2:172.21.15.124:6832/90278 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6b6c04c3c0 con 0x7f6b4c066f30 2023-10-06T01:42:37.385 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.384+0000 7f6b6b59e700 1 -- 172.21.15.27:0/1094155666 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6b4c066f30 msgr2=0x7f6b4c0693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:37.385 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.384+0000 7f6b6b59e700 1 --2- 172.21.15.27:0/1094155666 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6b4c066f30 0x7f6b4c0693e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6b64007fd0 tx=0x7f6b64007d30 comp rx=0 tx=0).stop 2023-10-06T01:42:37.385 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.384+0000 7f6b6b59e700 1 -- 172.21.15.27:0/1094155666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b6c159540 msgr2=0x7f6b6c16d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:37.385 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.384+0000 7f6b6b59e700 1 --2- 172.21.15.27:0/1094155666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b6c159540 0x7f6b6c16d3e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6b5400e900 tx=0x7f6b540091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:37.385 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.384+0000 7f6b6b59e700 1 -- 172.21.15.27:0/1094155666 shutdown_connections 2023-10-06T01:42:37.385 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.384+0000 7f6b6b59e700 1 --2- 172.21.15.27:0/1094155666 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6b4c066f30 0x7f6b4c0693e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:37.385 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.384+0000 7f6b6b59e700 1 --2- 172.21.15.27:0/1094155666 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6b6c0c9510 0x7f6b6c16cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:37.385 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.384+0000 7f6b6b59e700 1 --2- 172.21.15.27:0/1094155666 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6b6c159910 0x7f6b6c144420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:37.385 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.384+0000 7f6b6b59e700 1 --2- 172.21.15.27:0/1094155666 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6b6c159540 0x7f6b6c16d3e0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:37.385 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.384+0000 7f6b6b59e700 1 -- 172.21.15.27:0/1094155666 >> 172.21.15.27:0/1094155666 conn(0x7f6b6c06e600 msgr2=0x7f6b6c154d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:37.386 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.384+0000 7f6b6b59e700 1 -- 172.21.15.27:0/1094155666 shutdown_connections 2023-10-06T01:42:37.386 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.384+0000 7f6b6b59e700 1 -- 172.21.15.27:0/1094155666 wait complete. 2023-10-06T01:42:37.386 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 2 to deep-scrub 2023-10-06T01:42:37.396 INFO:tasks.ceph:Scrubbing osd.3 2023-10-06T01:42:37.396 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-10-06T01:42:37.520 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.520+0000 7fa296658700 1 Processor -- start 2023-10-06T01:42:37.520 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.520+0000 7fa296658700 1 -- start start 2023-10-06T01:42:37.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.521+0000 7fa296658700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fa2900b55b0 0x7fa2900ccbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:37.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.521+0000 7fa296658700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa2900d0f00 0x7fa2900ce270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:37.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.521+0000 7fa296658700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2900cbed0 con 0x7fa2900cf800 2023-10-06T01:42:37.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.521+0000 7fa296658700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa290159d40 con 0x7fa2900d0f00 2023-10-06T01:42:37.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.521+0000 7fa296658700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2900cc750 con 0x7fa2900b55b0 2023-10-06T01:42:37.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.521+0000 7fa294bf5700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa2900d0f00 0x7fa2900ce270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:37.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.521+0000 7fa28f7fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa2900cf800 0x7fa2900b5980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47374/0 (socket says 172.21.15.27:47374) 2023-10-06T01:42:37.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.521+0000 7fa28f7fe700 1 -- 172.21.15.27:0/1779427628 learned_addr learned my addr 172.21.15.27:0/1779427628 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:37.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.521+0000 7fa28ffff700 1 --2- 172.21.15.27:0/1779427628 >> v2:172.21.15.124:3300/0 conn(0x7fa2900b55b0 0x7fa2900ccbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:37.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.521+0000 7fa294bf5700 1 -- 172.21.15.27:0/1779427628 >> v2:172.21.15.124:3300/0 conn(0x7fa2900b55b0 msgr2=0x7fa2900ccbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:37.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.521+0000 7fa294bf5700 1 --2- 172.21.15.27:0/1779427628 >> v2:172.21.15.124:3300/0 conn(0x7fa2900b55b0 0x7fa2900ccbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:37.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.521+0000 7fa294bf5700 1 -- 172.21.15.27:0/1779427628 >> v1:172.21.15.27:6789/0 conn(0x7fa2900cf800 legacy=0x7fa2900b5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:37.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.521+0000 7fa294bf5700 1 -- 172.21.15.27:0/1779427628 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2900cffe0 con 0x7fa2900d0f00 2023-10-06T01:42:37.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.521+0000 7fa28effd700 1 -- 172.21.15.27:0/1779427628 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2105394029 0 0) 0x7fa2900cbed0 con 0x7fa2900cf800 2023-10-06T01:42:37.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.521+0000 7fa294bf5700 1 --2- 172.21.15.27:0/1779427628 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2900d0f00 0x7fa2900ce270 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa290089fe0 tx=0x7fa288009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de31e9f0c5276712 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:37.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.521+0000 7fa28effd700 1 -- 172.21.15.27:0/1779427628 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa28800e070 con 0x7fa2900d0f00 2023-10-06T01:42:37.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.521+0000 7fa28effd700 1 -- 172.21.15.27:0/1779427628 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa28800bc30 con 0x7fa2900d0f00 2023-10-06T01:42:37.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.521+0000 7fa28effd700 1 -- 172.21.15.27:0/1779427628 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa288012430 con 0x7fa2900d0f00 2023-10-06T01:42:37.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.522+0000 7fa296658700 1 -- 172.21.15.27:0/1779427628 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2900d0f00 msgr2=0x7fa2900ce270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:37.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.522+0000 7fa296658700 1 --2- 172.21.15.27:0/1779427628 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2900d0f00 0x7fa2900ce270 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa290089fe0 tx=0x7fa288009780 comp rx=0 tx=0).stop 2023-10-06T01:42:37.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.522+0000 7fa296658700 1 -- 172.21.15.27:0/1779427628 shutdown_connections 2023-10-06T01:42:37.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.522+0000 7fa296658700 1 --2- 172.21.15.27:0/1779427628 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2900d0f00 0x7fa2900ce270 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:37.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.522+0000 7fa296658700 1 --2- 172.21.15.27:0/1779427628 >> v2:172.21.15.124:3300/0 conn(0x7fa2900b55b0 0x7fa2900ccbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:37.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.522+0000 7fa296658700 1 -- 172.21.15.27:0/1779427628 >> 172.21.15.27:0/1779427628 conn(0x7fa29008a050 msgr2=0x7fa290099970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:37.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.522+0000 7fa296658700 1 -- 172.21.15.27:0/1779427628 shutdown_connections 2023-10-06T01:42:37.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.522+0000 7fa296658700 1 -- 172.21.15.27:0/1779427628 wait complete. 2023-10-06T01:42:37.527 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.523+0000 7fa296658700 1 Processor -- start 2023-10-06T01:42:37.527 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.523+0000 7fa296658700 1 -- start start 2023-10-06T01:42:37.527 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.524+0000 7fa296658700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa2900b55b0 0x7fa2900c6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:37.527 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.524+0000 7fa296658700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2900cf800 0x7fa2901490b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:37.527 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.524+0000 7fa296658700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa2900d0f00 0x7fa2900c5440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:37.527 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.524+0000 7fa296658700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa29016cc70 con 0x7fa2900cf800 2023-10-06T01:42:37.527 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.524+0000 7fa296658700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2900c6740 con 0x7fa2900d0f00 2023-10-06T01:42:37.527 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.524+0000 7fa296658700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa29016ce80 con 0x7fa2900b55b0 2023-10-06T01:42:37.527 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.524+0000 7fa28f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2900cf800 0x7fa2901490b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:37.527 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.524+0000 7fa28f7fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2900cf800 0x7fa2901490b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45878/0 (socket says 172.21.15.27:45878) 2023-10-06T01:42:37.527 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.524+0000 7fa28f7fe700 1 -- 172.21.15.27:0/4007258584 learned_addr learned my addr 172.21.15.27:0/4007258584 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:37.527 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.524+0000 7fa28ffff700 1 --2- 172.21.15.27:0/4007258584 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa2900b55b0 0x7fa2900c6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:37.527 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.524+0000 7fa294bf5700 1 --2- 172.21.15.27:0/4007258584 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa2900d0f00 0x7fa2900c5440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:37.528 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.524+0000 7fa28f7fe700 1 -- 172.21.15.27:0/4007258584 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa2900b55b0 msgr2=0x7fa2900c6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:37.528 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.524+0000 7fa28f7fe700 1 --2- 172.21.15.27:0/4007258584 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa2900b55b0 0x7fa2900c6200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:37.528 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.524+0000 7fa28f7fe700 1 -- 172.21.15.27:0/4007258584 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa2900d0f00 msgr2=0x7fa2900c5440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:37.528 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.524+0000 7fa28f7fe700 1 --2- 172.21.15.27:0/4007258584 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa2900d0f00 0x7fa2900c5440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:37.528 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.524+0000 7fa28f7fe700 1 -- 172.21.15.27:0/4007258584 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa288009a20 con 0x7fa2900cf800 2023-10-06T01:42:37.528 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.524+0000 7fa294bf5700 1 --2- 172.21.15.27:0/4007258584 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa2900d0f00 0x7fa2900c5440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:42:37.528 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.525+0000 7fa28f7fe700 1 --2- 172.21.15.27:0/4007258584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2900cf800 0x7fa2901490b0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fa27800c7e0 tx=0x7fa278007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:37.528 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.525+0000 7fa287fff700 1 -- 172.21.15.27:0/4007258584 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa278005b30 con 0x7fa2900cf800 2023-10-06T01:42:37.528 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.525+0000 7fa287fff700 1 -- 172.21.15.27:0/4007258584 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa278005c90 con 0x7fa2900cf800 2023-10-06T01:42:37.528 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.525+0000 7fa287fff700 1 -- 172.21.15.27:0/4007258584 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa278007300 con 0x7fa2900cf800 2023-10-06T01:42:37.528 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.525+0000 7fa296658700 1 -- 172.21.15.27:0/4007258584 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa29016d160 con 0x7fa2900cf800 2023-10-06T01:42:37.528 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.525+0000 7fa296658700 1 -- 172.21.15.27:0/4007258584 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa29016d720 con 0x7fa2900cf800 2023-10-06T01:42:37.529 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.525+0000 7fa296658700 1 -- 172.21.15.27:0/4007258584 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa2900cfd80 con 0x7fa2900cf800 2023-10-06T01:42:37.530 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.526+0000 7fa287fff700 1 -- 172.21.15.27:0/4007258584 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa27801b400 con 0x7fa2900cf800 2023-10-06T01:42:37.530 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.526+0000 7fa287fff700 1 --2- 172.21.15.27:0/4007258584 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa270066f30 0x7fa2700693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:37.530 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.527+0000 7fa287fff700 1 -- 172.21.15.27:0/4007258584 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5292+0+0 (secure 0 0 0) 0x7fa278083f10 con 0x7fa2900cf800 2023-10-06T01:42:37.530 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.527+0000 7fa28ffff700 1 --2- 172.21.15.27:0/4007258584 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa270066f30 0x7fa2700693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:37.530 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.527+0000 7fa287fff700 1 --2- 172.21.15.27:0/4007258584 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x7fa27006d3a0 0x7fa27006f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:37.530 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.527+0000 7fa287fff700 1 -- 172.21.15.27:0/4007258584 --> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa27006fe80 con 0x7fa27006d3a0 2023-10-06T01:42:37.530 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.527+0000 7fa287fff700 1 -- 172.21.15.27:0/4007258584 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa278086550 con 0x7fa2900cf800 2023-10-06T01:42:37.530 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.527+0000 7fa294bf5700 1 --2- 172.21.15.27:0/4007258584 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x7fa27006d3a0 0x7fa27006f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:37.530 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.527+0000 7fa294bf5700 1 --2- 172.21.15.27:0/4007258584 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x7fa27006d3a0 0x7fa27006f7b0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:37.530 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.527+0000 7fa28ffff700 1 --2- 172.21.15.27:0/4007258584 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa270066f30 0x7fa2700693e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa280006fd0 tx=0x7fa280008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:37.531 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.528+0000 7fa287fff700 1 -- 172.21.15.27:0/4007258584 <== osd.3 v2:172.21.15.27:6824/90668 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fa27006fe80 con 0x7fa27006d3a0 2023-10-06T01:42:37.542 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.542+0000 7fa296658700 1 -- 172.21.15.27:0/4007258584 --> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fa2900717c0 con 0x7fa27006d3a0 2023-10-06T01:42:37.542 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.542+0000 7fa287fff700 1 -- 172.21.15.27:0/4007258584 <== osd.3 v2:172.21.15.27:6824/90668 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fa2900717c0 con 0x7fa27006d3a0 2023-10-06T01:42:37.543 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2023-10-06T01:42:37.543 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-06T01:42:37.543 INFO:teuthology.orchestra.run.smithi027.stdout:} 2023-10-06T01:42:37.544 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.542+0000 7fa296658700 1 -- 172.21.15.27:0/4007258584 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x7fa27006d3a0 msgr2=0x7fa27006f7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:37.544 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.542+0000 7fa296658700 1 --2- 172.21.15.27:0/4007258584 >> [v2:172.21.15.27:6824/90668,v1:172.21.15.27:6825/90668] conn(0x7fa27006d3a0 0x7fa27006f7b0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:37.544 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.542+0000 7fa296658700 1 -- 172.21.15.27:0/4007258584 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa270066f30 msgr2=0x7fa2700693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:37.544 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.542+0000 7fa296658700 1 --2- 172.21.15.27:0/4007258584 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa270066f30 0x7fa2700693e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fa280006fd0 tx=0x7fa280008040 comp rx=0 tx=0).stop 2023-10-06T01:42:37.544 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.542+0000 7fa296658700 1 -- 172.21.15.27:0/4007258584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2900cf800 msgr2=0x7fa2901490b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:37.544 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.542+0000 7fa296658700 1 --2- 172.21.15.27:0/4007258584 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa2900cf800 0x7fa2901490b0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fa27800c7e0 tx=0x7fa278007df0 comp rx=0 tx=0).stop 2023-10-06T01:42:37.544 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.542+0000 7fa294bf5700 1 -- 172.21.15.27:0/4007258584 reap_dead start 2023-10-06T01:42:37.544 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.542+0000 7fa296658700 1 -- 172.21.15.27:0/4007258584 shutdown_connections 2023-10-06T01:42:37.544 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.542+0000 7fa296658700 1 -- 172.21.15.27:0/4007258584 >> 172.21.15.27:0/4007258584 conn(0x7fa29008a050 msgr2=0x7fa2901592b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:37.544 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.543+0000 7fa296658700 1 -- 172.21.15.27:0/4007258584 shutdown_connections 2023-10-06T01:42:37.545 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.543+0000 7fa296658700 1 -- 172.21.15.27:0/4007258584 wait complete. 2023-10-06T01:42:37.554 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-10-06T01:42:37.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.675+0000 7f9f64945700 1 Processor -- start 2023-10-06T01:42:37.676 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.675+0000 7f9f64945700 1 -- start start 2023-10-06T01:42:37.677 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.676+0000 7f9f64945700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9f601598d0 0x7f9f6015d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:37.677 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.676+0000 7f9f64945700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9f6015d8e0 0x7f9f600cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:37.677 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.676+0000 7f9f64945700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f6015dd20 con 0x7f9f60159500 2023-10-06T01:42:37.677 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.676+0000 7f9f64945700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f600b1110 con 0x7f9f601598d0 2023-10-06T01:42:37.678 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.676+0000 7f9f64945700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f600b13b0 con 0x7f9f6015d8e0 2023-10-06T01:42:37.678 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.676+0000 7f9f5dd9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9f601598d0 0x7f9f6015d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:37.678 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.676+0000 7f9f5dd9b700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f9f601598d0 0x7f9f6015d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45888/0 (socket says 172.21.15.27:45888) 2023-10-06T01:42:37.678 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.676+0000 7f9f5dd9b700 1 -- 172.21.15.27:0/4171627135 learned_addr learned my addr 172.21.15.27:0/4171627135 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:37.678 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.676+0000 7f9f5ed9d700 1 --2- 172.21.15.27:0/4171627135 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9f6015d8e0 0x7f9f600cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:37.678 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.677+0000 7f9f5dd9b700 1 -- 172.21.15.27:0/4171627135 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9f6015d8e0 msgr2=0x7f9f600cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:37.679 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.677+0000 7f9f5dd9b700 1 --2- 172.21.15.27:0/4171627135 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9f6015d8e0 0x7f9f600cf6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:37.679 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.677+0000 7f9f5dd9b700 1 -- 172.21.15.27:0/4171627135 >> v1:172.21.15.124:6789/0 conn(0x7f9f60159500 legacy=0x7f9f600b3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:37.679 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.677+0000 7f9f5dd9b700 1 -- 172.21.15.27:0/4171627135 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f600b1690 con 0x7f9f601598d0 2023-10-06T01:42:37.679 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.677+0000 7f9f5dd9b700 1 --2- 172.21.15.27:0/4171627135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f601598d0 0x7f9f6015d3a0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f9f480099b0 tx=0x7f9f48009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac8ec4d2efff9c2f server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:37.679 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.677+0000 7f9f5d59a700 1 -- 172.21.15.27:0/4171627135 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f48009ea0 con 0x7f9f601598d0 2023-10-06T01:42:37.679 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.677+0000 7f9f5d59a700 1 -- 172.21.15.27:0/4171627135 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f4800b860 con 0x7f9f601598d0 2023-10-06T01:42:37.679 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.677+0000 7f9f5d59a700 1 -- 172.21.15.27:0/4171627135 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4800bc00 con 0x7f9f601598d0 2023-10-06T01:42:37.683 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.677+0000 7f9f64945700 1 -- 172.21.15.27:0/4171627135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f601598d0 msgr2=0x7f9f6015d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:37.683 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.677+0000 7f9f64945700 1 --2- 172.21.15.27:0/4171627135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f601598d0 0x7f9f6015d3a0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f9f480099b0 tx=0x7f9f48009250 comp rx=0 tx=0).stop 2023-10-06T01:42:37.683 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.678+0000 7f9f64945700 1 -- 172.21.15.27:0/4171627135 shutdown_connections 2023-10-06T01:42:37.683 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.678+0000 7f9f64945700 1 --2- 172.21.15.27:0/4171627135 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9f6015d8e0 0x7f9f600cf6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:37.683 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.678+0000 7f9f64945700 1 --2- 172.21.15.27:0/4171627135 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f601598d0 0x7f9f6015d3a0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:37.683 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.678+0000 7f9f64945700 1 -- 172.21.15.27:0/4171627135 >> 172.21.15.27:0/4171627135 conn(0x7f9f60067630 msgr2=0x7f9f60069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:37.683 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.678+0000 7f9f64945700 1 -- 172.21.15.27:0/4171627135 shutdown_connections 2023-10-06T01:42:37.683 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.678+0000 7f9f64945700 1 -- 172.21.15.27:0/4171627135 wait complete. 2023-10-06T01:42:37.683 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.679+0000 7f9f64945700 1 Processor -- start 2023-10-06T01:42:37.683 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.679+0000 7f9f64945700 1 -- start start 2023-10-06T01:42:37.684 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.680+0000 7f9f64945700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f60159500 0x7f9f60152df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:37.684 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.680+0000 7f9f64945700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9f601598d0 0x7f9f60150a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:37.684 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.680+0000 7f9f64945700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9f6015d8e0 0x7f9f6014bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:37.684 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.680+0000 7f9f64945700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f600b5510 con 0x7f9f60159500 2023-10-06T01:42:37.684 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.680+0000 7f9f64945700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f600b57f0 con 0x7f9f601598d0 2023-10-06T01:42:37.684 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.680+0000 7f9f64945700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f600b5ad0 con 0x7f9f6015d8e0 2023-10-06T01:42:37.684 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.680+0000 7f9f5e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f60159500 0x7f9f60152df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:37.685 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.680+0000 7f9f5e59c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f60159500 0x7f9f60152df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45898/0 (socket says 172.21.15.27:45898) 2023-10-06T01:42:37.685 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.680+0000 7f9f5e59c700 1 -- 172.21.15.27:0/803965907 learned_addr learned my addr 172.21.15.27:0/803965907 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:37.685 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.680+0000 7f9f5ed9d700 1 --2- 172.21.15.27:0/803965907 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9f6015d8e0 0x7f9f6014bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:37.685 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.680+0000 7f9f5dd9b700 1 --2- 172.21.15.27:0/803965907 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9f601598d0 0x7f9f60150a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:37.685 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.680+0000 7f9f5e59c700 1 -- 172.21.15.27:0/803965907 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9f6015d8e0 msgr2=0x7f9f6014bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:37.685 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.680+0000 7f9f5e59c700 1 --2- 172.21.15.27:0/803965907 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9f6015d8e0 0x7f9f6014bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:37.685 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.680+0000 7f9f5e59c700 1 -- 172.21.15.27:0/803965907 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9f601598d0 msgr2=0x7f9f60150a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:37.685 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.680+0000 7f9f5e59c700 1 --2- 172.21.15.27:0/803965907 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9f601598d0 0x7f9f60150a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:37.685 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.680+0000 7f9f5e59c700 1 -- 172.21.15.27:0/803965907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f480094f0 con 0x7f9f60159500 2023-10-06T01:42:37.686 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.680+0000 7f9f5dd9b700 1 --2- 172.21.15.27:0/803965907 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9f601598d0 0x7f9f60150a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:42:37.686 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.680+0000 7f9f5e59c700 1 --2- 172.21.15.27:0/803965907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f60159500 0x7f9f60152df0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f9f5800c590 tx=0x7f9f5800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:37.686 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.680+0000 7f9f567fc700 1 -- 172.21.15.27:0/803965907 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f58011590 con 0x7f9f60159500 2023-10-06T01:42:37.686 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.680+0000 7f9f567fc700 1 -- 172.21.15.27:0/803965907 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f58011710 con 0x7f9f60159500 2023-10-06T01:42:37.686 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.680+0000 7f9f567fc700 1 -- 172.21.15.27:0/803965907 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f58011ad0 con 0x7f9f60159500 2023-10-06T01:42:37.686 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.680+0000 7f9f64945700 1 -- 172.21.15.27:0/803965907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f600b5db0 con 0x7f9f60159500 2023-10-06T01:42:37.686 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.680+0000 7f9f64945700 1 -- 172.21.15.27:0/803965907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f60153330 con 0x7f9f60159500 2023-10-06T01:42:37.687 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.681+0000 7f9f64945700 1 -- 172.21.15.27:0/803965907 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f6014dd90 con 0x7f9f60159500 2023-10-06T01:42:37.687 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.682+0000 7f9f567fc700 1 -- 172.21.15.27:0/803965907 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f9f5801b3b0 con 0x7f9f60159500 2023-10-06T01:42:37.687 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.682+0000 7f9f567fc700 1 --2- 172.21.15.27:0/803965907 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9f40066f30 0x7f9f400693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:37.687 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.682+0000 7f9f567fc700 1 -- 172.21.15.27:0/803965907 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5292+0+0 (secure 0 0 0) 0x7f9f580838c0 con 0x7f9f60159500 2023-10-06T01:42:37.687 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.682+0000 7f9f5dd9b700 1 --2- 172.21.15.27:0/803965907 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9f40066f30 0x7f9f400693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:37.687 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.683+0000 7f9f5dd9b700 1 --2- 172.21.15.27:0/803965907 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9f40066f30 0x7f9f400693e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9f480099b0 tx=0x7f9f48017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:37.688 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.686+0000 7f9f567fc700 1 -- 172.21.15.27:0/803965907 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9f58050cd0 con 0x7f9f60159500 2023-10-06T01:42:37.879 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.879+0000 7f9f64945700 1 -- 172.21.15.27:0/803965907 --> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f9f60152760 con 0x7f9f40066f30 2023-10-06T01:42:37.880 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.880+0000 7f9f567fc700 1 -- 172.21.15.27:0/803965907 <== mgr.4100 v2:172.21.15.124:6832/90278 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f9f60152760 con 0x7f9f40066f30 2023-10-06T01:42:37.882 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.882+0000 7f9f64945700 1 -- 172.21.15.27:0/803965907 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9f40066f30 msgr2=0x7f9f400693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:37.883 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.882+0000 7f9f64945700 1 --2- 172.21.15.27:0/803965907 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9f40066f30 0x7f9f400693e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f9f480099b0 tx=0x7f9f48017000 comp rx=0 tx=0).stop 2023-10-06T01:42:37.883 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.882+0000 7f9f64945700 1 -- 172.21.15.27:0/803965907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f60159500 msgr2=0x7f9f60152df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:37.883 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.882+0000 7f9f64945700 1 --2- 172.21.15.27:0/803965907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f60159500 0x7f9f60152df0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f9f5800c590 tx=0x7f9f5800c8a0 comp rx=0 tx=0).stop 2023-10-06T01:42:37.883 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.882+0000 7f9f64945700 1 -- 172.21.15.27:0/803965907 shutdown_connections 2023-10-06T01:42:37.883 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.882+0000 7f9f64945700 1 --2- 172.21.15.27:0/803965907 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f9f40066f30 0x7f9f400693e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:37.883 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.882+0000 7f9f64945700 1 --2- 172.21.15.27:0/803965907 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f9f6015d8e0 0x7f9f6014bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:37.883 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.882+0000 7f9f64945700 1 --2- 172.21.15.27:0/803965907 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f9f60159500 0x7f9f60152df0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:37.883 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.882+0000 7f9f64945700 1 --2- 172.21.15.27:0/803965907 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f9f601598d0 0x7f9f60150a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:37.883 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.882+0000 7f9f64945700 1 -- 172.21.15.27:0/803965907 >> 172.21.15.27:0/803965907 conn(0x7f9f60067630 msgr2=0x7f9f60158410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:37.883 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.882+0000 7f9f64945700 1 -- 172.21.15.27:0/803965907 shutdown_connections 2023-10-06T01:42:37.883 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:37.882+0000 7f9f64945700 1 -- 172.21.15.27:0/803965907 wait complete. 2023-10-06T01:42:37.884 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 3 to deep-scrub 2023-10-06T01:42:37.894 INFO:tasks.ceph:Scrubbing osd.4 2023-10-06T01:42:37.894 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-10-06T01:42:38.016 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.016+0000 7fb50d6e5700 1 Processor -- start 2023-10-06T01:42:38.016 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.016+0000 7fb50d6e5700 1 -- start start 2023-10-06T01:42:38.018 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.017+0000 7fb50d6e5700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb5080ccbe0 0x7fb5080b5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:38.019 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.017+0000 7fb50d6e5700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5080cbed0 con 0x7fb5080b55b0 2023-10-06T01:42:38.019 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.017+0000 7fb50d6e5700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5080cc750 con 0x7fb5080d0f00 2023-10-06T01:42:38.019 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.017+0000 7fb50d6e5700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb508159d40 con 0x7fb5080ccbe0 2023-10-06T01:42:38.019 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.017+0000 7fb506ffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fb5080b55b0 0x7fb5080cf800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47378/0 (socket says 172.21.15.27:47378) 2023-10-06T01:42:38.019 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.017+0000 7fb506ffd700 1 -- 172.21.15.27:0/1163207039 learned_addr learned my addr 172.21.15.27:0/1163207039 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:38.019 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.017+0000 7fb5067fc700 1 --2- 172.21.15.27:0/1163207039 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb5080ccbe0 0x7fb5080b5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:38.019 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.017+0000 7fb505ffb700 1 -- 172.21.15.27:0/1163207039 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2010469998 0 0) 0x7fb5080cbed0 con 0x7fb5080b55b0 2023-10-06T01:42:38.019 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.017+0000 7fb505ffb700 1 -- 172.21.15.27:0/1163207039 --> v1:172.21.15.27:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4f0003570 con 0x7fb5080b55b0 2023-10-06T01:42:38.019 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.017+0000 7fb505ffb700 1 -- 172.21.15.27:0/1163207039 <== mon.1 v1:172.21.15.124:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1029119235 0 0) 0x7fb5080cc750 con 0x7fb5080d0f00 2023-10-06T01:42:38.019 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.017+0000 7fb505ffb700 1 -- 172.21.15.27:0/1163207039 --> v1:172.21.15.124:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5080cbed0 con 0x7fb5080d0f00 2023-10-06T01:42:38.020 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.017+0000 7fb505ffb700 1 -- 172.21.15.27:0/1163207039 <== mon.0 v1:172.21.15.27:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1345804578 0 0) 0x7fb4f0003570 con 0x7fb5080b55b0 2023-10-06T01:42:38.020 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.017+0000 7fb5067fc700 1 -- 172.21.15.27:0/1163207039 >> v1:172.21.15.124:6789/0 conn(0x7fb5080d0f00 legacy=0x7fb5080ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:38.020 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.017+0000 7fb5067fc700 1 -- 172.21.15.27:0/1163207039 >> v1:172.21.15.27:6789/0 conn(0x7fb5080b55b0 legacy=0x7fb5080cf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:38.020 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.017+0000 7fb5067fc700 1 -- 172.21.15.27:0/1163207039 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5080cd050 con 0x7fb5080ccbe0 2023-10-06T01:42:38.020 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.018+0000 7fb5067fc700 1 --2- 172.21.15.27:0/1163207039 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb5080ccbe0 0x7fb5080b5980 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc009910 tx=0x7fb4fc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23914c4e220ca054 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:38.020 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.018+0000 7fb505ffb700 1 -- 172.21.15.27:0/1163207039 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f80015f0 con 0x7fb5080ccbe0 2023-10-06T01:42:38.020 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.018+0000 7fb505ffb700 1 -- 172.21.15.27:0/1163207039 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4fc011420 con 0x7fb5080ccbe0 2023-10-06T01:42:38.020 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.018+0000 7fb505ffb700 1 -- 172.21.15.27:0/1163207039 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4fc0117c0 con 0x7fb5080ccbe0 2023-10-06T01:42:38.020 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.018+0000 7fb50d6e5700 1 -- 172.21.15.27:0/1163207039 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb5080ccbe0 msgr2=0x7fb5080b5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:38.020 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.018+0000 7fb50d6e5700 1 --2- 172.21.15.27:0/1163207039 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb5080ccbe0 0x7fb5080b5980 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb4fc009910 tx=0x7fb4fc0091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:38.020 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.018+0000 7fb50d6e5700 1 -- 172.21.15.27:0/1163207039 shutdown_connections 2023-10-06T01:42:38.020 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.018+0000 7fb50d6e5700 1 --2- 172.21.15.27:0/1163207039 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb5080ccbe0 0x7fb5080b5980 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:38.021 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.018+0000 7fb50d6e5700 1 -- 172.21.15.27:0/1163207039 >> 172.21.15.27:0/1163207039 conn(0x7fb50808a050 msgr2=0x7fb508099970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:38.021 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.018+0000 7fb50d6e5700 1 -- 172.21.15.27:0/1163207039 shutdown_connections 2023-10-06T01:42:38.021 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.018+0000 7fb50d6e5700 1 -- 172.21.15.27:0/1163207039 wait complete. 2023-10-06T01:42:38.024 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.020+0000 7fb50d6e5700 1 Processor -- start 2023-10-06T01:42:38.024 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.020+0000 7fb50d6e5700 1 -- start start 2023-10-06T01:42:38.024 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.020+0000 7fb50d6e5700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb5080b55b0 0x7fb508156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:38.024 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.020+0000 7fb50d6e5700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb5080ccbe0 0x7fb5080c5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:38.024 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.020+0000 7fb50d6e5700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5080d0f00 0x7fb5080b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:38.024 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.020+0000 7fb50d6e5700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb508068950 con 0x7fb5080d0f00 2023-10-06T01:42:38.024 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.020+0000 7fb50d6e5700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb508069790 con 0x7fb5080ccbe0 2023-10-06T01:42:38.024 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.020+0000 7fb50d6e5700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb508068b80 con 0x7fb5080b55b0 2023-10-06T01:42:38.024 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.020+0000 7fb5077fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5080d0f00 0x7fb5080b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:38.024 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.020+0000 7fb5077fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5080d0f00 0x7fb5080b2100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45912/0 (socket says 172.21.15.27:45912) 2023-10-06T01:42:38.025 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.021+0000 7fb5077fe700 1 -- 172.21.15.27:0/2515706310 learned_addr learned my addr 172.21.15.27:0/2515706310 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:38.025 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.021+0000 7fb506ffd700 1 --2- 172.21.15.27:0/2515706310 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb5080b55b0 0x7fb508156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:38.025 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.021+0000 7fb5067fc700 1 --2- 172.21.15.27:0/2515706310 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb5080ccbe0 0x7fb5080c5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:38.025 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.021+0000 7fb5077fe700 1 -- 172.21.15.27:0/2515706310 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb5080b55b0 msgr2=0x7fb508156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:38.025 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.021+0000 7fb5077fe700 1 --2- 172.21.15.27:0/2515706310 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fb5080b55b0 0x7fb508156dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:38.025 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.021+0000 7fb5077fe700 1 -- 172.21.15.27:0/2515706310 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb5080ccbe0 msgr2=0x7fb5080c5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:38.025 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.021+0000 7fb5077fe700 1 --2- 172.21.15.27:0/2515706310 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb5080ccbe0 0x7fb5080c5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:38.025 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.021+0000 7fb5077fe700 1 -- 172.21.15.27:0/2515706310 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4fc009420 con 0x7fb5080d0f00 2023-10-06T01:42:38.025 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.021+0000 7fb5067fc700 1 --2- 172.21.15.27:0/2515706310 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fb5080ccbe0 0x7fb5080c5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:42:38.025 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.021+0000 7fb5077fe700 1 --2- 172.21.15.27:0/2515706310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5080d0f00 0x7fb5080b2100 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fb50000c860 tx=0x7fb500002700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:38.025 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.021+0000 7fb4eeffd700 1 -- 172.21.15.27:0/2515706310 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb500002030 con 0x7fb5080d0f00 2023-10-06T01:42:38.025 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.021+0000 7fb4eeffd700 1 -- 172.21.15.27:0/2515706310 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5000021b0 con 0x7fb5080d0f00 2023-10-06T01:42:38.026 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.021+0000 7fb50d6e5700 1 -- 172.21.15.27:0/2515706310 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb508068e60 con 0x7fb5080d0f00 2023-10-06T01:42:38.026 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.021+0000 7fb4eeffd700 1 -- 172.21.15.27:0/2515706310 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb500012630 con 0x7fb5080d0f00 2023-10-06T01:42:38.026 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.021+0000 7fb50d6e5700 1 -- 172.21.15.27:0/2515706310 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb508069420 con 0x7fb5080d0f00 2023-10-06T01:42:38.026 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.022+0000 7fb50d6e5700 1 -- 172.21.15.27:0/2515706310 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb5080717c0 con 0x7fb5080d0f00 2023-10-06T01:42:38.026 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.023+0000 7fb4eeffd700 1 -- 172.21.15.27:0/2515706310 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fb500012a20 con 0x7fb5080d0f00 2023-10-06T01:42:38.026 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.023+0000 7fb4eeffd700 1 --2- 172.21.15.27:0/2515706310 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb4f4066f30 0x7fb4f40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:38.026 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.023+0000 7fb4eeffd700 1 -- 172.21.15.27:0/2515706310 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5292+0+0 (secure 0 0 0) 0x7fb500084330 con 0x7fb5080d0f00 2023-10-06T01:42:38.026 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.023+0000 7fb506ffd700 1 --2- 172.21.15.27:0/2515706310 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb4f4066f30 0x7fb4f40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:38.026 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.023+0000 7fb4eeffd700 1 --2- 172.21.15.27:0/2515706310 >> [v2:172.21.15.124:6800/90863,v1:172.21.15.124:6801/90863] conn(0x7fb4f406d3a0 0x7fb4f406f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:38.026 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.023+0000 7fb4eeffd700 1 -- 172.21.15.27:0/2515706310 --> [v2:172.21.15.124:6800/90863,v1:172.21.15.124:6801/90863] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb4f406fe80 con 0x7fb4f406d3a0 2023-10-06T01:42:38.026 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.023+0000 7fb4eeffd700 1 -- 172.21.15.27:0/2515706310 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb500086970 con 0x7fb5080d0f00 2023-10-06T01:42:38.026 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.023+0000 7fb5067fc700 1 --2- 172.21.15.27:0/2515706310 >> [v2:172.21.15.124:6800/90863,v1:172.21.15.124:6801/90863] conn(0x7fb4f406d3a0 0x7fb4f406f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:38.026 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.024+0000 7fb506ffd700 1 --2- 172.21.15.27:0/2515706310 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb4f4066f30 0x7fb4f40693e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb4f8006d80 tx=0x7fb4f80012c0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:38.027 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.024+0000 7fb5067fc700 1 --2- 172.21.15.27:0/2515706310 >> [v2:172.21.15.124:6800/90863,v1:172.21.15.124:6801/90863] conn(0x7fb4f406d3a0 0x7fb4f406f7b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:38.027 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.025+0000 7fb4eeffd700 1 -- 172.21.15.27:0/2515706310 <== osd.4 v2:172.21.15.124:6800/90863 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fb4f406fe80 con 0x7fb4f406d3a0 2023-10-06T01:42:38.038 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.038+0000 7fb50d6e5700 1 -- 172.21.15.27:0/2515706310 --> [v2:172.21.15.124:6800/90863,v1:172.21.15.124:6801/90863] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb5080ad170 con 0x7fb4f406d3a0 2023-10-06T01:42:38.039 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.039+0000 7fb4eeffd700 1 -- 172.21.15.27:0/2515706310 <== osd.4 v2:172.21.15.124:6800/90863 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fb5080ad170 con 0x7fb4f406d3a0 2023-10-06T01:42:38.039 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2023-10-06T01:42:38.039 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-06T01:42:38.039 INFO:teuthology.orchestra.run.smithi027.stdout:} 2023-10-06T01:42:38.040 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.039+0000 7fb50d6e5700 1 -- 172.21.15.27:0/2515706310 >> [v2:172.21.15.124:6800/90863,v1:172.21.15.124:6801/90863] conn(0x7fb4f406d3a0 msgr2=0x7fb4f406f7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:38.040 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.039+0000 7fb50d6e5700 1 --2- 172.21.15.27:0/2515706310 >> [v2:172.21.15.124:6800/90863,v1:172.21.15.124:6801/90863] conn(0x7fb4f406d3a0 0x7fb4f406f7b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:38.040 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.039+0000 7fb50d6e5700 1 -- 172.21.15.27:0/2515706310 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb4f4066f30 msgr2=0x7fb4f40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:38.041 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.039+0000 7fb50d6e5700 1 --2- 172.21.15.27:0/2515706310 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fb4f4066f30 0x7fb4f40693e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb4f8006d80 tx=0x7fb4f80012c0 comp rx=0 tx=0).stop 2023-10-06T01:42:38.041 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.039+0000 7fb50d6e5700 1 -- 172.21.15.27:0/2515706310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5080d0f00 msgr2=0x7fb5080b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:38.041 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.039+0000 7fb50d6e5700 1 --2- 172.21.15.27:0/2515706310 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fb5080d0f00 0x7fb5080b2100 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fb50000c860 tx=0x7fb500002700 comp rx=0 tx=0).stop 2023-10-06T01:42:38.041 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.039+0000 7fb5077fe700 1 -- 172.21.15.27:0/2515706310 reap_dead start 2023-10-06T01:42:38.041 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.039+0000 7fb50d6e5700 1 -- 172.21.15.27:0/2515706310 shutdown_connections 2023-10-06T01:42:38.041 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.039+0000 7fb50d6e5700 1 -- 172.21.15.27:0/2515706310 >> 172.21.15.27:0/2515706310 conn(0x7fb50808a050 msgr2=0x7fb5081590f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:38.041 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.039+0000 7fb50d6e5700 1 -- 172.21.15.27:0/2515706310 shutdown_connections 2023-10-06T01:42:38.041 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.039+0000 7fb50d6e5700 1 -- 172.21.15.27:0/2515706310 wait complete. 2023-10-06T01:42:38.051 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-10-06T01:42:38.170 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.170+0000 7fa1aa197700 1 Processor -- start 2023-10-06T01:42:38.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.170+0000 7fa1aa197700 1 -- start start 2023-10-06T01:42:38.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.171+0000 7fa1aa197700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa1a415de80 0x7fa1a40b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:38.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.171+0000 7fa1aa197700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fa1a40b61d0 0x7fa1a415d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:38.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.171+0000 7fa1aa197700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a4069380 con 0x7fa1a4142e90 2023-10-06T01:42:38.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.171+0000 7fa1aa197700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1a4069f20 con 0x7fa1a415de80 2023-10-06T01:42:38.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.171+0000 7fa1aa197700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1a4068770 con 0x7fa1a40b61d0 2023-10-06T01:42:38.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.171+0000 7fa1a8994700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa1a415de80 0x7fa1a40b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:38.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.171+0000 7fa1a9195700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7fa1a4142e90 0x7fa1a4069810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47382/0 (socket says 172.21.15.27:47382) 2023-10-06T01:42:38.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.171+0000 7fa1a9195700 1 -- 172.21.15.27:0/3088422326 learned_addr learned my addr 172.21.15.27:0/3088422326 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:38.173 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.172+0000 7fa1a8994700 1 -- 172.21.15.27:0/3088422326 >> v2:172.21.15.124:3300/0 conn(0x7fa1a40b61d0 msgr2=0x7fa1a415d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:38.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.172+0000 7fa1a9996700 1 --2- 172.21.15.27:0/3088422326 >> v2:172.21.15.124:3300/0 conn(0x7fa1a40b61d0 0x7fa1a415d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:38.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.172+0000 7fa1a8994700 1 --2- 172.21.15.27:0/3088422326 >> v2:172.21.15.124:3300/0 conn(0x7fa1a40b61d0 0x7fa1a415d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:38.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.172+0000 7fa1a8994700 1 -- 172.21.15.27:0/3088422326 >> v1:172.21.15.27:6789/0 conn(0x7fa1a4142e90 legacy=0x7fa1a4069810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:38.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.172+0000 7fa1a3fff700 1 -- 172.21.15.27:0/3088422326 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1251100015 0 0) 0x7fa1a4069380 con 0x7fa1a4142e90 2023-10-06T01:42:38.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.172+0000 7fa1a8994700 1 -- 172.21.15.27:0/3088422326 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a4068a50 con 0x7fa1a415de80 2023-10-06T01:42:38.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.172+0000 7fa1a8994700 1 --2- 172.21.15.27:0/3088422326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1a415de80 0x7fa1a40b5c90 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fa1a40d0540 tx=0x7fa1980091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b839488090c4e40 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:38.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.172+0000 7fa1a3fff700 1 -- 172.21.15.27:0/3088422326 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19800b6b0 con 0x7fa1a415de80 2023-10-06T01:42:38.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.172+0000 7fa1a3fff700 1 -- 172.21.15.27:0/3088422326 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa19800b810 con 0x7fa1a415de80 2023-10-06T01:42:38.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.172+0000 7fa1a3fff700 1 -- 172.21.15.27:0/3088422326 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19800bbb0 con 0x7fa1a415de80 2023-10-06T01:42:38.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.172+0000 7fa1aa197700 1 -- 172.21.15.27:0/3088422326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1a415de80 msgr2=0x7fa1a40b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:38.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.172+0000 7fa1aa197700 1 --2- 172.21.15.27:0/3088422326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1a415de80 0x7fa1a40b5c90 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fa1a40d0540 tx=0x7fa1980091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:38.175 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.172+0000 7fa1aa197700 1 -- 172.21.15.27:0/3088422326 shutdown_connections 2023-10-06T01:42:38.175 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.173+0000 7fa1aa197700 1 --2- 172.21.15.27:0/3088422326 >> v2:172.21.15.124:3300/0 conn(0x7fa1a40b61d0 0x7fa1a415d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:38.175 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.173+0000 7fa1aa197700 1 --2- 172.21.15.27:0/3088422326 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1a415de80 0x7fa1a40b5c90 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:38.175 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.173+0000 7fa1aa197700 1 -- 172.21.15.27:0/3088422326 >> 172.21.15.27:0/3088422326 conn(0x7fa1a40ccfe0 msgr2=0x7fa1a40cd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:38.175 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.173+0000 7fa1aa197700 1 -- 172.21.15.27:0/3088422326 shutdown_connections 2023-10-06T01:42:38.175 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.173+0000 7fa1aa197700 1 -- 172.21.15.27:0/3088422326 wait complete. 2023-10-06T01:42:38.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.174+0000 7fa1aa197700 1 Processor -- start 2023-10-06T01:42:38.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.174+0000 7fa1aa197700 1 -- start start 2023-10-06T01:42:38.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.175+0000 7fa1aa197700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa1a40b61d0 0x7fa1a416d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:38.177 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.175+0000 7fa1aa197700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa1a4142e90 0x7fa1a40c53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:38.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.175+0000 7fa1aa197700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1a415de80 0x7fa1a40c8090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:38.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.175+0000 7fa1aa197700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a40c5930 con 0x7fa1a415de80 2023-10-06T01:42:38.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.175+0000 7fa1aa197700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a40c5d70 con 0x7fa1a40b61d0 2023-10-06T01:42:38.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.175+0000 7fa1aa197700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a416d830 con 0x7fa1a4142e90 2023-10-06T01:42:38.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.175+0000 7fa1a9996700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1a415de80 0x7fa1a40c8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:38.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.175+0000 7fa1a9996700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1a415de80 0x7fa1a40c8090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45928/0 (socket says 172.21.15.27:45928) 2023-10-06T01:42:38.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.175+0000 7fa1a9996700 1 -- 172.21.15.27:0/285807397 learned_addr learned my addr 172.21.15.27:0/285807397 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:38.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.175+0000 7fa1a8994700 1 --2- 172.21.15.27:0/285807397 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa1a4142e90 0x7fa1a40c53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:38.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.175+0000 7fa1a9195700 1 --2- 172.21.15.27:0/285807397 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa1a40b61d0 0x7fa1a416d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:38.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.175+0000 7fa1a9996700 1 -- 172.21.15.27:0/285807397 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa1a4142e90 msgr2=0x7fa1a40c53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:38.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.175+0000 7fa1a9996700 1 --2- 172.21.15.27:0/285807397 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa1a4142e90 0x7fa1a40c53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:38.178 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.175+0000 7fa1a9996700 1 -- 172.21.15.27:0/285807397 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa1a40b61d0 msgr2=0x7fa1a416d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:38.179 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.175+0000 7fa1a9996700 1 --2- 172.21.15.27:0/285807397 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa1a40b61d0 0x7fa1a416d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:38.179 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.175+0000 7fa1a9996700 1 -- 172.21.15.27:0/285807397 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa198009420 con 0x7fa1a415de80 2023-10-06T01:42:38.179 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.175+0000 7fa1a9195700 1 --2- 172.21.15.27:0/285807397 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa1a40b61d0 0x7fa1a416d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:42:38.179 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.175+0000 7fa1a9996700 1 --2- 172.21.15.27:0/285807397 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1a415de80 0x7fa1a40c8090 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fa19c00c880 tx=0x7fa19c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:38.179 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.175+0000 7fa1a17fa700 1 -- 172.21.15.27:0/285807397 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19c011600 con 0x7fa1a415de80 2023-10-06T01:42:38.179 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.175+0000 7fa1a17fa700 1 -- 172.21.15.27:0/285807397 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa19c011760 con 0x7fa1a415de80 2023-10-06T01:42:38.179 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.175+0000 7fa1aa197700 1 -- 172.21.15.27:0/285807397 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1a416db10 con 0x7fa1a415de80 2023-10-06T01:42:38.179 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.175+0000 7fa1a17fa700 1 -- 172.21.15.27:0/285807397 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa19c011ae0 con 0x7fa1a415de80 2023-10-06T01:42:38.179 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.176+0000 7fa1aa197700 1 -- 172.21.15.27:0/285807397 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1a40c85d0 con 0x7fa1a415de80 2023-10-06T01:42:38.179 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.176+0000 7fa1aa197700 1 -- 172.21.15.27:0/285807397 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1a40c5b40 con 0x7fa1a415de80 2023-10-06T01:42:38.179 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.177+0000 7fa1a17fa700 1 -- 172.21.15.27:0/285807397 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa19c0193f0 con 0x7fa1a415de80 2023-10-06T01:42:38.179 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.177+0000 7fa1a17fa700 1 --2- 172.21.15.27:0/285807397 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa190066f30 0x7fa1900693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:38.180 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.177+0000 7fa1a17fa700 1 -- 172.21.15.27:0/285807397 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5292+0+0 (secure 0 0 0) 0x7fa19c084770 con 0x7fa1a415de80 2023-10-06T01:42:38.180 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.178+0000 7fa1a9195700 1 --2- 172.21.15.27:0/285807397 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa190066f30 0x7fa1900693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:38.180 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.178+0000 7fa1a9195700 1 --2- 172.21.15.27:0/285807397 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa190066f30 0x7fa1900693e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa194001560 tx=0x7fa194009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:38.181 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.181+0000 7fa1a17fa700 1 -- 172.21.15.27:0/285807397 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa1a40c5b40 con 0x7fa1a415de80 2023-10-06T01:42:38.372 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.372+0000 7fa1aa197700 1 -- 172.21.15.27:0/285807397 --> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fa1a40c5b40 con 0x7fa190066f30 2023-10-06T01:42:38.373 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.373+0000 7fa1a17fa700 1 -- 172.21.15.27:0/285807397 <== mgr.4100 v2:172.21.15.124:6832/90278 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa1a40c5b40 con 0x7fa190066f30 2023-10-06T01:42:38.377 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.375+0000 7fa1aa197700 1 -- 172.21.15.27:0/285807397 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa190066f30 msgr2=0x7fa1900693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:38.377 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.375+0000 7fa1aa197700 1 --2- 172.21.15.27:0/285807397 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa190066f30 0x7fa1900693e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa194001560 tx=0x7fa194009040 comp rx=0 tx=0).stop 2023-10-06T01:42:38.377 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.375+0000 7fa1aa197700 1 -- 172.21.15.27:0/285807397 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1a415de80 msgr2=0x7fa1a40c8090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:38.378 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.375+0000 7fa1aa197700 1 --2- 172.21.15.27:0/285807397 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1a415de80 0x7fa1a40c8090 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fa19c00c880 tx=0x7fa19c008040 comp rx=0 tx=0).stop 2023-10-06T01:42:38.378 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.375+0000 7fa1aa197700 1 -- 172.21.15.27:0/285807397 shutdown_connections 2023-10-06T01:42:38.378 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.375+0000 7fa1aa197700 1 --2- 172.21.15.27:0/285807397 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa190066f30 0x7fa1900693e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:38.378 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.375+0000 7fa1aa197700 1 --2- 172.21.15.27:0/285807397 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa1a415de80 0x7fa1a40c8090 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:38.378 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.375+0000 7fa1aa197700 1 --2- 172.21.15.27:0/285807397 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa1a40b61d0 0x7fa1a416d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:38.378 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.375+0000 7fa1aa197700 1 --2- 172.21.15.27:0/285807397 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa1a4142e90 0x7fa1a40c53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:38.378 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.375+0000 7fa1aa197700 1 -- 172.21.15.27:0/285807397 >> 172.21.15.27:0/285807397 conn(0x7fa1a40ccfe0 msgr2=0x7fa1a41407c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:38.378 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.375+0000 7fa1aa197700 1 -- 172.21.15.27:0/285807397 shutdown_connections 2023-10-06T01:42:38.378 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.375+0000 7fa1aa197700 1 -- 172.21.15.27:0/285807397 wait complete. 2023-10-06T01:42:38.379 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 4 to deep-scrub 2023-10-06T01:42:38.387 INFO:tasks.ceph:Scrubbing osd.5 2023-10-06T01:42:38.387 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-10-06T01:42:38.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.509+0000 7f21e359e700 1 Processor -- start 2023-10-06T01:42:38.510 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.509+0000 7f21e359e700 1 -- start start 2023-10-06T01:42:38.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.510+0000 7f21e359e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f21e40cf810 0x7f21e40cc850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:38.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.510+0000 7f21e359e700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f21e40cbb00 0x7f21e4158e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:38.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.510+0000 7f21e359e700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21e40cb670 con 0x7f21e40cf440 2023-10-06T01:42:38.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.510+0000 7f21e359e700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21e40cbf60 con 0x7f21e40cbb00 2023-10-06T01:42:38.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.510+0000 7f21e359e700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21e40cce90 con 0x7f21e40cf810 2023-10-06T01:42:38.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.511+0000 7f21e1d9b700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f21e40cf810 0x7f21e40cc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:38.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.511+0000 7f21e2d9d700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f21e40cbb00 0x7f21e4158e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:38.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.511+0000 7f21e1d9b700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f21e40cf810 0x7f21e40cc850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.27:45106/0 (socket says 172.21.15.27:45106) 2023-10-06T01:42:38.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.511+0000 7f21e1d9b700 1 -- 172.21.15.27:0/941394398 learned_addr learned my addr 172.21.15.27:0/941394398 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:38.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.511+0000 7f21e1d9b700 1 -- 172.21.15.27:0/941394398 >> v2:172.21.15.124:3300/0 conn(0x7f21e40cbb00 msgr2=0x7f21e4158e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:38.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.511+0000 7f21e1d9b700 1 --2- 172.21.15.27:0/941394398 >> v2:172.21.15.124:3300/0 conn(0x7f21e40cbb00 0x7f21e4158e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:38.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.511+0000 7f21e1d9b700 1 -- 172.21.15.27:0/941394398 >> v1:172.21.15.124:6789/0 conn(0x7f21e40cf440 legacy=0x7f21e40cc140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:38.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.511+0000 7f21e1d9b700 1 -- 172.21.15.27:0/941394398 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e40cad80 con 0x7f21e40cf810 2023-10-06T01:42:38.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.511+0000 7f21e1d9b700 1 --2- 172.21.15.27:0/941394398 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f21e40cf810 0x7f21e40cc850 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f21cc0099b0 tx=0x7f21cc009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a6263c2d97ef244 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:38.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.511+0000 7f21e159a700 1 -- 172.21.15.27:0/941394398 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21cc009ea0 con 0x7f21e40cf810 2023-10-06T01:42:38.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.512+0000 7f21e159a700 1 -- 172.21.15.27:0/941394398 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21cc00b860 con 0x7f21e40cf810 2023-10-06T01:42:38.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.512+0000 7f21e159a700 1 -- 172.21.15.27:0/941394398 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21cc00bc00 con 0x7f21e40cf810 2023-10-06T01:42:38.515 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.512+0000 7f21e359e700 1 -- 172.21.15.27:0/941394398 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f21e40cf810 msgr2=0x7f21e40cc850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:38.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.512+0000 7f21e359e700 1 --2- 172.21.15.27:0/941394398 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f21e40cf810 0x7f21e40cc850 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f21cc0099b0 tx=0x7f21cc009250 comp rx=0 tx=0).stop 2023-10-06T01:42:38.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.512+0000 7f21e359e700 1 -- 172.21.15.27:0/941394398 shutdown_connections 2023-10-06T01:42:38.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.512+0000 7f21e359e700 1 --2- 172.21.15.27:0/941394398 >> v2:172.21.15.124:3300/0 conn(0x7f21e40cbb00 0x7f21e4158e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:38.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.512+0000 7f21e359e700 1 --2- 172.21.15.27:0/941394398 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f21e40cf810 0x7f21e40cc850 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:38.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.512+0000 7f21e359e700 1 -- 172.21.15.27:0/941394398 >> 172.21.15.27:0/941394398 conn(0x7f21e40cd080 msgr2=0x7f21e40cd480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:38.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.512+0000 7f21e359e700 1 -- 172.21.15.27:0/941394398 shutdown_connections 2023-10-06T01:42:38.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.512+0000 7f21e359e700 1 -- 172.21.15.27:0/941394398 wait complete. 2023-10-06T01:42:38.516 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.513+0000 7f21e359e700 1 Processor -- start 2023-10-06T01:42:38.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.513+0000 7f21e359e700 1 -- start start 2023-10-06T01:42:38.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.514+0000 7f21e359e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f21e40cbb00 0x7f21e40c82c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:38.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.514+0000 7f21e359e700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f21e40cf440 0x7f21e40b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:38.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.514+0000 7f21e359e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21e40cf810 0x7f21e415af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:38.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.514+0000 7f21e359e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e40c8800 con 0x7f21e40cf810 2023-10-06T01:42:38.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.514+0000 7f21e359e700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21e416a000 con 0x7f21e40cf440 2023-10-06T01:42:38.517 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.514+0000 7f21e359e700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21e416a2a0 con 0x7f21e40cbb00 2023-10-06T01:42:38.518 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.514+0000 7f21e2d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21e40cf810 0x7f21e415af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:38.521 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.514+0000 7f21e259c700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f21e40cbb00 0x7f21e40c82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:38.521 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.514+0000 7f21e2d9d700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21e40cf810 0x7f21e415af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45934/0 (socket says 172.21.15.27:45934) 2023-10-06T01:42:38.521 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.514+0000 7f21e259c700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f21e40cbb00 0x7f21e40c82c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.124:3301/0 says I am v2:172.21.15.27:45110/0 (socket says 172.21.15.27:45110) 2023-10-06T01:42:38.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.514+0000 7f21e2d9d700 1 -- 172.21.15.27:0/1481615643 learned_addr learned my addr 172.21.15.27:0/1481615643 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:38.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.514+0000 7f21e1d9b700 1 --2- 172.21.15.27:0/1481615643 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f21e40cf440 0x7f21e40b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:38.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.515+0000 7f21e259c700 1 -- 172.21.15.27:0/1481615643 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f21e40cf440 msgr2=0x7f21e40b4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:38.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.515+0000 7f21e259c700 1 --2- 172.21.15.27:0/1481615643 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f21e40cf440 0x7f21e40b4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:38.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.515+0000 7f21e259c700 1 -- 172.21.15.27:0/1481615643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21e40cf810 msgr2=0x7f21e415af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:38.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.515+0000 7f21e259c700 1 --2- 172.21.15.27:0/1481615643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21e40cf810 0x7f21e415af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:38.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.515+0000 7f21e259c700 1 -- 172.21.15.27:0/1481615643 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21cc0094f0 con 0x7f21e40cbb00 2023-10-06T01:42:38.522 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.515+0000 7f21e2d9d700 1 --2- 172.21.15.27:0/1481615643 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f21e40cf810 0x7f21e415af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-06T01:42:38.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.515+0000 7f21e259c700 1 --2- 172.21.15.27:0/1481615643 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f21e40cbb00 0x7f21e40c82c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f21d400c910 tx=0x7f21d40016f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:38.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.515+0000 7f21da7fc700 1 -- 172.21.15.27:0/1481615643 <== mon.2 v2:172.21.15.124:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d4012630 con 0x7f21e40cbb00 2023-10-06T01:42:38.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.515+0000 7f21da7fc700 1 -- 172.21.15.27:0/1481615643 <== mon.2 v2:172.21.15.124:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21d40127b0 con 0x7f21e40cbb00 2023-10-06T01:42:38.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.515+0000 7f21e359e700 1 -- 172.21.15.27:0/1481615643 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21e416a580 con 0x7f21e40cbb00 2023-10-06T01:42:38.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.515+0000 7f21da7fc700 1 -- 172.21.15.27:0/1481615643 <== mon.2 v2:172.21.15.124:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21d4012b70 con 0x7f21e40cbb00 2023-10-06T01:42:38.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.515+0000 7f21e359e700 1 -- 172.21.15.27:0/1481615643 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21e416ab40 con 0x7f21e40cbb00 2023-10-06T01:42:38.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.516+0000 7f21e359e700 1 -- 172.21.15.27:0/1481615643 --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f21e4069490 con 0x7f21e40cbb00 2023-10-06T01:42:38.523 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.516+0000 7f21da7fc700 1 -- 172.21.15.27:0/1481615643 <== mon.2 v2:172.21.15.124:3301/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f21d4003030 con 0x7f21e40cbb00 2023-10-06T01:42:38.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.517+0000 7f21da7fc700 1 --2- 172.21.15.27:0/1481615643 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f21c4066f30 0x7f21c40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:38.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.517+0000 7f21da7fc700 1 -- 172.21.15.27:0/1481615643 <== mon.2 v2:172.21.15.124:3301/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5292+0+0 (secure 0 0 0) 0x7f21d40849d0 con 0x7f21e40cbb00 2023-10-06T01:42:38.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.517+0000 7f21da7fc700 1 --2- 172.21.15.27:0/1481615643 >> [v2:172.21.15.124:6816/90885,v1:172.21.15.124:6817/90885] conn(0x7f21c406d3a0 0x7f21c406f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:38.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.517+0000 7f21da7fc700 1 -- 172.21.15.27:0/1481615643 --> [v2:172.21.15.124:6816/90885,v1:172.21.15.124:6817/90885] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f21c406fe80 con 0x7f21c406d3a0 2023-10-06T01:42:38.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.517+0000 7f21da7fc700 1 -- 172.21.15.27:0/1481615643 <== mon.2 v2:172.21.15.124:3301/0 6 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21d4084da0 con 0x7f21e40cbb00 2023-10-06T01:42:38.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.517+0000 7f21e1d9b700 1 --2- 172.21.15.27:0/1481615643 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f21c4066f30 0x7f21c40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:38.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.517+0000 7f21e2d9d700 1 --2- 172.21.15.27:0/1481615643 >> [v2:172.21.15.124:6816/90885,v1:172.21.15.124:6817/90885] conn(0x7f21c406d3a0 0x7f21c406f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:38.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.518+0000 7f21e1d9b700 1 --2- 172.21.15.27:0/1481615643 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f21c4066f30 0x7f21c40693e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f21cc0099b0 tx=0x7f21cc017000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:38.524 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.518+0000 7f21e2d9d700 1 --2- 172.21.15.27:0/1481615643 >> [v2:172.21.15.124:6816/90885,v1:172.21.15.124:6817/90885] conn(0x7f21c406d3a0 0x7f21c406f7b0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:38.525 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.519+0000 7f21da7fc700 1 -- 172.21.15.27:0/1481615643 <== osd.5 v2:172.21.15.124:6816/90885 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f21c406fe80 con 0x7f21c406d3a0 2023-10-06T01:42:38.533 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.533+0000 7f21e359e700 1 -- 172.21.15.27:0/1481615643 --> [v2:172.21.15.124:6816/90885,v1:172.21.15.124:6817/90885] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f21e416a3e0 con 0x7f21c406d3a0 2023-10-06T01:42:38.533 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.533+0000 7f21da7fc700 1 -- 172.21.15.27:0/1481615643 <== osd.5 v2:172.21.15.124:6816/90885 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f21e416a3e0 con 0x7f21c406d3a0 2023-10-06T01:42:38.534 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2023-10-06T01:42:38.534 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-06T01:42:38.534 INFO:teuthology.orchestra.run.smithi027.stdout:} 2023-10-06T01:42:38.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.533+0000 7f21e359e700 1 -- 172.21.15.27:0/1481615643 >> [v2:172.21.15.124:6816/90885,v1:172.21.15.124:6817/90885] conn(0x7f21c406d3a0 msgr2=0x7f21c406f7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:38.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.533+0000 7f21e359e700 1 --2- 172.21.15.27:0/1481615643 >> [v2:172.21.15.124:6816/90885,v1:172.21.15.124:6817/90885] conn(0x7f21c406d3a0 0x7f21c406f7b0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:38.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.533+0000 7f21e359e700 1 -- 172.21.15.27:0/1481615643 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f21c4066f30 msgr2=0x7f21c40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:38.535 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.533+0000 7f21e359e700 1 --2- 172.21.15.27:0/1481615643 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f21c4066f30 0x7f21c40693e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f21cc0099b0 tx=0x7f21cc017000 comp rx=0 tx=0).stop 2023-10-06T01:42:38.536 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.533+0000 7f21e359e700 1 -- 172.21.15.27:0/1481615643 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f21e40cbb00 msgr2=0x7f21e40c82c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:38.536 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.533+0000 7f21e359e700 1 --2- 172.21.15.27:0/1481615643 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f21e40cbb00 0x7f21e40c82c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f21d400c910 tx=0x7f21d40016f0 comp rx=0 tx=0).stop 2023-10-06T01:42:38.536 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.533+0000 7f21e2d9d700 1 -- 172.21.15.27:0/1481615643 reap_dead start 2023-10-06T01:42:38.536 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.533+0000 7f21e359e700 1 -- 172.21.15.27:0/1481615643 shutdown_connections 2023-10-06T01:42:38.536 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.533+0000 7f21e359e700 1 -- 172.21.15.27:0/1481615643 >> 172.21.15.27:0/1481615643 conn(0x7f21e40cd080 msgr2=0x7f21e40c53b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:38.536 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.533+0000 7f21e359e700 1 -- 172.21.15.27:0/1481615643 shutdown_connections 2023-10-06T01:42:38.536 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.533+0000 7f21e359e700 1 -- 172.21.15.27:0/1481615643 wait complete. 2023-10-06T01:42:38.545 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-10-06T01:42:38.667 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.666+0000 7fc9b0152700 1 Processor -- start 2023-10-06T01:42:38.667 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.667+0000 7fc9b0152700 1 -- start start 2023-10-06T01:42:38.668 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.667+0000 7fc9b0152700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc9a81598d0 0x7fc9a815d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:38.668 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.667+0000 7fc9b0152700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fc9a815d8e0 0x7fc9a80cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:38.669 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.667+0000 7fc9b0152700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9a815dd20 con 0x7fc9a8159500 2023-10-06T01:42:38.669 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.667+0000 7fc9b0152700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9a80b1110 con 0x7fc9a81598d0 2023-10-06T01:42:38.669 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.667+0000 7fc9b0152700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9a80b13b0 con 0x7fc9a815d8e0 2023-10-06T01:42:38.669 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.667+0000 7fc9ad6ed700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc9a81598d0 0x7fc9a815d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:38.669 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.668+0000 7fc9ad6ed700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fc9a81598d0 0x7fc9a815d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45948/0 (socket says 172.21.15.27:45948) 2023-10-06T01:42:38.669 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.668+0000 7fc9ad6ed700 1 -- 172.21.15.27:0/3221650312 learned_addr learned my addr 172.21.15.27:0/3221650312 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:38.669 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.668+0000 7fc9ae6ef700 1 --2- 172.21.15.27:0/3221650312 >> v2:172.21.15.124:3300/0 conn(0x7fc9a815d8e0 0x7fc9a80cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:38.669 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.668+0000 7fc9ad6ed700 1 -- 172.21.15.27:0/3221650312 >> v2:172.21.15.124:3300/0 conn(0x7fc9a815d8e0 msgr2=0x7fc9a80cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:38.669 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.668+0000 7fc9ad6ed700 1 --2- 172.21.15.27:0/3221650312 >> v2:172.21.15.124:3300/0 conn(0x7fc9a815d8e0 0x7fc9a80cf6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:38.669 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.668+0000 7fc9ad6ed700 1 -- 172.21.15.27:0/3221650312 >> v1:172.21.15.124:6789/0 conn(0x7fc9a8159500 legacy=0x7fc9a80b3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:38.669 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.668+0000 7fc9ad6ed700 1 -- 172.21.15.27:0/3221650312 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9a80b1690 con 0x7fc9a81598d0 2023-10-06T01:42:38.670 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.668+0000 7fc9ad6ed700 1 --2- 172.21.15.27:0/3221650312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9a81598d0 0x7fc9a815d3a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fc9a806a270 tx=0x7fc99c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d95ad768d0806ecb server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:38.670 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.668+0000 7fc9aceec700 1 -- 172.21.15.27:0/3221650312 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99c009e70 con 0x7fc9a81598d0 2023-10-06T01:42:38.670 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.668+0000 7fc9aceec700 1 -- 172.21.15.27:0/3221650312 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc99c00b860 con 0x7fc9a81598d0 2023-10-06T01:42:38.670 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.668+0000 7fc9aceec700 1 -- 172.21.15.27:0/3221650312 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99c00bc00 con 0x7fc9a81598d0 2023-10-06T01:42:38.673 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.669+0000 7fc9b0152700 1 -- 172.21.15.27:0/3221650312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9a81598d0 msgr2=0x7fc9a815d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:38.673 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.669+0000 7fc9b0152700 1 --2- 172.21.15.27:0/3221650312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9a81598d0 0x7fc9a815d3a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fc9a806a270 tx=0x7fc99c009250 comp rx=0 tx=0).stop 2023-10-06T01:42:38.673 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.669+0000 7fc9b0152700 1 -- 172.21.15.27:0/3221650312 shutdown_connections 2023-10-06T01:42:38.673 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.669+0000 7fc9b0152700 1 --2- 172.21.15.27:0/3221650312 >> v2:172.21.15.124:3300/0 conn(0x7fc9a815d8e0 0x7fc9a80cf6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:38.673 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.669+0000 7fc9b0152700 1 --2- 172.21.15.27:0/3221650312 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9a81598d0 0x7fc9a815d3a0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:38.673 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.669+0000 7fc9b0152700 1 -- 172.21.15.27:0/3221650312 >> 172.21.15.27:0/3221650312 conn(0x7fc9a8067630 msgr2=0x7fc9a8069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:38.673 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.669+0000 7fc9b0152700 1 -- 172.21.15.27:0/3221650312 shutdown_connections 2023-10-06T01:42:38.673 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.669+0000 7fc9b0152700 1 -- 172.21.15.27:0/3221650312 wait complete. 2023-10-06T01:42:38.673 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.670+0000 7fc9b0152700 1 Processor -- start 2023-10-06T01:42:38.673 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.670+0000 7fc9b0152700 1 -- start start 2023-10-06T01:42:38.674 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.671+0000 7fc9b0152700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc9a8159500 0x7fc9a815c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:38.674 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.671+0000 7fc9b0152700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc9a81598d0 0x7fc9a814b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:38.674 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.671+0000 7fc9b0152700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9a815d8e0 0x7fc9a814b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:38.674 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.671+0000 7fc9b0152700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a815aa20 con 0x7fc9a815d8e0 2023-10-06T01:42:38.674 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.671+0000 7fc9b0152700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a816e3c0 con 0x7fc9a81598d0 2023-10-06T01:42:38.674 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.671+0000 7fc9b0152700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a816b030 con 0x7fc9a8159500 2023-10-06T01:42:38.674 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.671+0000 7fc9ae6ef700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9a815d8e0 0x7fc9a814b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:38.674 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.671+0000 7fc9ae6ef700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9a815d8e0 0x7fc9a814b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45954/0 (socket says 172.21.15.27:45954) 2023-10-06T01:42:38.674 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.671+0000 7fc9ae6ef700 1 -- 172.21.15.27:0/1474957868 learned_addr learned my addr 172.21.15.27:0/1474957868 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:38.674 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.671+0000 7fc9adeee700 1 --2- 172.21.15.27:0/1474957868 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc9a8159500 0x7fc9a815c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:38.674 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.671+0000 7fc9ad6ed700 1 --2- 172.21.15.27:0/1474957868 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc9a81598d0 0x7fc9a814b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:38.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.671+0000 7fc9ae6ef700 1 -- 172.21.15.27:0/1474957868 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc9a8159500 msgr2=0x7fc9a815c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:38.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.671+0000 7fc9ae6ef700 1 --2- 172.21.15.27:0/1474957868 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc9a8159500 0x7fc9a815c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:38.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.671+0000 7fc9ae6ef700 1 -- 172.21.15.27:0/1474957868 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc9a81598d0 msgr2=0x7fc9a814b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:38.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.671+0000 7fc9ae6ef700 1 --2- 172.21.15.27:0/1474957868 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc9a81598d0 0x7fc9a814b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:38.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.671+0000 7fc9ae6ef700 1 -- 172.21.15.27:0/1474957868 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc99c0094f0 con 0x7fc9a815d8e0 2023-10-06T01:42:38.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.671+0000 7fc9ae6ef700 1 --2- 172.21.15.27:0/1474957868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9a815d8e0 0x7fc9a814b780 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fc998009e20 tx=0x7fc99800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:38.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.671+0000 7fc9a5ffb700 1 -- 172.21.15.27:0/1474957868 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99800caa0 con 0x7fc9a815d8e0 2023-10-06T01:42:38.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.672+0000 7fc9a5ffb700 1 -- 172.21.15.27:0/1474957868 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc99800f040 con 0x7fc9a815d8e0 2023-10-06T01:42:38.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.672+0000 7fc9b0152700 1 -- 172.21.15.27:0/1474957868 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9a816d600 con 0x7fc9a815d8e0 2023-10-06T01:42:38.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.672+0000 7fc9a5ffb700 1 -- 172.21.15.27:0/1474957868 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99800ce70 con 0x7fc9a815d8e0 2023-10-06T01:42:38.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.672+0000 7fc9b0152700 1 -- 172.21.15.27:0/1474957868 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9a816dbc0 con 0x7fc9a815d8e0 2023-10-06T01:42:38.675 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.673+0000 7fc9b0152700 1 -- 172.21.15.27:0/1474957868 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9a8064050 con 0x7fc9a815d8e0 2023-10-06T01:42:38.676 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.673+0000 7fc9a5ffb700 1 -- 172.21.15.27:0/1474957868 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fc998014690 con 0x7fc9a815d8e0 2023-10-06T01:42:38.676 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.674+0000 7fc9a5ffb700 1 --2- 172.21.15.27:0/1474957868 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc994066f30 0x7fc9940693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:38.676 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.674+0000 7fc9a5ffb700 1 -- 172.21.15.27:0/1474957868 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5292+0+0 (secure 0 0 0) 0x7fc998084c20 con 0x7fc9a815d8e0 2023-10-06T01:42:38.676 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.674+0000 7fc9adeee700 1 --2- 172.21.15.27:0/1474957868 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc994066f30 0x7fc9940693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:38.677 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.674+0000 7fc9adeee700 1 --2- 172.21.15.27:0/1474957868 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc994066f30 0x7fc9940693e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc9a806fde0 tx=0x7fc9a0009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:38.678 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.678+0000 7fc9a5ffb700 1 -- 172.21.15.27:0/1474957868 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc998051f90 con 0x7fc9a815d8e0 2023-10-06T01:42:38.871 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.871+0000 7fc9b0152700 1 -- 172.21.15.27:0/1474957868 --> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fc9a80b1970 con 0x7fc994066f30 2023-10-06T01:42:38.872 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.872+0000 7fc9a5ffb700 1 -- 172.21.15.27:0/1474957868 <== mgr.4100 v2:172.21.15.124:6832/90278 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fc9a80b1970 con 0x7fc994066f30 2023-10-06T01:42:38.874 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.874+0000 7fc9b0152700 1 -- 172.21.15.27:0/1474957868 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc994066f30 msgr2=0x7fc9940693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:38.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.874+0000 7fc9b0152700 1 --2- 172.21.15.27:0/1474957868 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc994066f30 0x7fc9940693e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc9a806fde0 tx=0x7fc9a0009040 comp rx=0 tx=0).stop 2023-10-06T01:42:38.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.874+0000 7fc9b0152700 1 -- 172.21.15.27:0/1474957868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9a815d8e0 msgr2=0x7fc9a814b780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:38.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.874+0000 7fc9b0152700 1 --2- 172.21.15.27:0/1474957868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9a815d8e0 0x7fc9a814b780 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fc998009e20 tx=0x7fc99800b040 comp rx=0 tx=0).stop 2023-10-06T01:42:38.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.874+0000 7fc9b0152700 1 -- 172.21.15.27:0/1474957868 shutdown_connections 2023-10-06T01:42:38.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.874+0000 7fc9b0152700 1 --2- 172.21.15.27:0/1474957868 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fc994066f30 0x7fc9940693e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:38.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.874+0000 7fc9b0152700 1 --2- 172.21.15.27:0/1474957868 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fc9a8159500 0x7fc9a815c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:38.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.874+0000 7fc9b0152700 1 --2- 172.21.15.27:0/1474957868 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fc9a815d8e0 0x7fc9a814b780 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:38.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.874+0000 7fc9b0152700 1 --2- 172.21.15.27:0/1474957868 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fc9a81598d0 0x7fc9a814b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:38.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.874+0000 7fc9b0152700 1 -- 172.21.15.27:0/1474957868 >> 172.21.15.27:0/1474957868 conn(0x7fc9a8067630 msgr2=0x7fc9a80ceb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:38.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.874+0000 7fc9b0152700 1 -- 172.21.15.27:0/1474957868 shutdown_connections 2023-10-06T01:42:38.875 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:38.874+0000 7fc9b0152700 1 -- 172.21.15.27:0/1474957868 wait complete. 2023-10-06T01:42:38.876 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 5 to deep-scrub 2023-10-06T01:42:38.886 INFO:tasks.ceph:Scrubbing osd.6 2023-10-06T01:42:38.886 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-10-06T01:42:39.009 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.009+0000 7f6779d4c700 1 Processor -- start 2023-10-06T01:42:39.009 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.009+0000 7f6779d4c700 1 -- start start 2023-10-06T01:42:39.011 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.009+0000 7f6779d4c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f67740b3940 0x7f67740b3d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:39.011 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.009+0000 7f6779d4c700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6774159100 0x7f6774159550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:39.011 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.010+0000 7f6779d4c700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6774069a10 con 0x7f677415d450 2023-10-06T01:42:39.011 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.010+0000 7f6779d4c700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f677406a590 con 0x7f67740b3940 2023-10-06T01:42:39.011 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.010+0000 7f6779d4c700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6774069cf0 con 0x7f6774159100 2023-10-06T01:42:39.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.010+0000 7f67737fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f67740b3940 0x7f67740b3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:39.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.010+0000 7f6772ffd700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f677415d450 0x7f67740691e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47396/0 (socket says 172.21.15.27:47396) 2023-10-06T01:42:39.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.010+0000 7f6772ffd700 1 -- 172.21.15.27:0/2779825113 learned_addr learned my addr 172.21.15.27:0/2779825113 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:39.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.010+0000 7f6773fff700 1 --2- 172.21.15.27:0/2779825113 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6774159100 0x7f6774159550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:39.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.010+0000 7f67737fe700 1 -- 172.21.15.27:0/2779825113 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6774159100 msgr2=0x7f6774159550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:39.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.010+0000 7f67737fe700 1 --2- 172.21.15.27:0/2779825113 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6774159100 0x7f6774159550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:39.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.010+0000 7f67737fe700 1 -- 172.21.15.27:0/2779825113 >> v1:172.21.15.27:6789/0 conn(0x7f677415d450 legacy=0x7f67740691e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:39.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.010+0000 7f67727fc700 1 -- 172.21.15.27:0/2779825113 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2975327890 0 0) 0x7f6774069a10 con 0x7f677415d450 2023-10-06T01:42:39.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.010+0000 7f67737fe700 1 -- 172.21.15.27:0/2779825113 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6774069fd0 con 0x7f67740b3940 2023-10-06T01:42:39.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.010+0000 7f67737fe700 1 --2- 172.21.15.27:0/2779825113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67740b3940 0x7f67740b3d10 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f67640098e0 tx=0x7f67640091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cacf9c79dd1909ca server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:39.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.010+0000 7f67727fc700 1 -- 172.21.15.27:0/2779825113 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676400b640 con 0x7f67740b3940 2023-10-06T01:42:39.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.010+0000 7f67727fc700 1 -- 172.21.15.27:0/2779825113 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f676400b7a0 con 0x7f67740b3940 2023-10-06T01:42:39.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.010+0000 7f67727fc700 1 -- 172.21.15.27:0/2779825113 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676400bb40 con 0x7f67740b3940 2023-10-06T01:42:39.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.011+0000 7f6779d4c700 1 -- 172.21.15.27:0/2779825113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67740b3940 msgr2=0x7f67740b3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:39.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.011+0000 7f6779d4c700 1 --2- 172.21.15.27:0/2779825113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67740b3940 0x7f67740b3d10 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f67640098e0 tx=0x7f67640091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:39.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.011+0000 7f6779d4c700 1 -- 172.21.15.27:0/2779825113 shutdown_connections 2023-10-06T01:42:39.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.011+0000 7f6779d4c700 1 --2- 172.21.15.27:0/2779825113 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6774159100 0x7f6774159550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:39.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.011+0000 7f6779d4c700 1 --2- 172.21.15.27:0/2779825113 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67740b3940 0x7f67740b3d10 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:39.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.011+0000 7f6779d4c700 1 -- 172.21.15.27:0/2779825113 >> 172.21.15.27:0/2779825113 conn(0x7f67740ce4c0 msgr2=0x7f67740ce890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:39.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.011+0000 7f6779d4c700 1 -- 172.21.15.27:0/2779825113 shutdown_connections 2023-10-06T01:42:39.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.011+0000 7f6779d4c700 1 -- 172.21.15.27:0/2779825113 wait complete. 2023-10-06T01:42:39.016 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.012+0000 7f6779d4c700 1 Processor -- start 2023-10-06T01:42:39.016 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.012+0000 7f6779d4c700 1 -- start start 2023-10-06T01:42:39.016 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.013+0000 7f6779d4c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67740b3940 0x7f6774147000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:39.016 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.013+0000 7f6779d4c700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6774159100 0x7f67740c7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:39.017 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.013+0000 7f6779d4c700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f677415d450 0x7f677406b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:39.017 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.013+0000 7f6779d4c700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6774147540 con 0x7f67740b3940 2023-10-06T01:42:39.017 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.013+0000 7f6779d4c700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6774151190 con 0x7f677415d450 2023-10-06T01:42:39.017 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.013+0000 7f6779d4c700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f677415bf60 con 0x7f6774159100 2023-10-06T01:42:39.017 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.013+0000 7f67737fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67740b3940 0x7f6774147000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:39.017 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.013+0000 7f67737fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67740b3940 0x7f6774147000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45980/0 (socket says 172.21.15.27:45980) 2023-10-06T01:42:39.017 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.013+0000 7f67737fe700 1 -- 172.21.15.27:0/234863999 learned_addr learned my addr 172.21.15.27:0/234863999 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:39.017 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.013+0000 7f6772ffd700 1 --2- 172.21.15.27:0/234863999 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6774159100 0x7f67740c7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:39.017 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.013+0000 7f6773fff700 1 --2- 172.21.15.27:0/234863999 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f677415d450 0x7f677406b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:39.017 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.013+0000 7f67737fe700 1 -- 172.21.15.27:0/234863999 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6774159100 msgr2=0x7f67740c7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:39.017 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.013+0000 7f67737fe700 1 --2- 172.21.15.27:0/234863999 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6774159100 0x7f67740c7f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:39.017 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.013+0000 7f67737fe700 1 -- 172.21.15.27:0/234863999 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f677415d450 msgr2=0x7f677406b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:39.017 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.013+0000 7f67737fe700 1 --2- 172.21.15.27:0/234863999 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f677415d450 0x7f677406b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:39.018 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.013+0000 7f67737fe700 1 -- 172.21.15.27:0/234863999 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6764009420 con 0x7f67740b3940 2023-10-06T01:42:39.018 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.013+0000 7f6772ffd700 1 --2- 172.21.15.27:0/234863999 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6774159100 0x7f67740c7f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:42:39.018 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.013+0000 7f67737fe700 1 --2- 172.21.15.27:0/234863999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67740b3940 0x7f6774147000 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f6764009f80 tx=0x7f6764017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:39.018 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.014+0000 7f675b7fe700 1 -- 172.21.15.27:0/234863999 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676400bb20 con 0x7f67740b3940 2023-10-06T01:42:39.018 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.014+0000 7f675b7fe700 1 -- 172.21.15.27:0/234863999 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6764011420 con 0x7f67740b3940 2023-10-06T01:42:39.018 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.014+0000 7f6779d4c700 1 -- 172.21.15.27:0/234863999 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f677415b350 con 0x7f67740b3940 2023-10-06T01:42:39.018 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.014+0000 7f675b7fe700 1 -- 172.21.15.27:0/234863999 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67640117a0 con 0x7f67740b3940 2023-10-06T01:42:39.018 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.014+0000 7f6779d4c700 1 -- 172.21.15.27:0/234863999 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f677415b910 con 0x7f67740b3940 2023-10-06T01:42:39.018 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.014+0000 7f6779d4c700 1 -- 172.21.15.27:0/234863999 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f67740717c0 con 0x7f67740b3940 2023-10-06T01:42:39.018 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.015+0000 7f675b7fe700 1 -- 172.21.15.27:0/234863999 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6764011b50 con 0x7f67740b3940 2023-10-06T01:42:39.018 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.015+0000 7f675b7fe700 1 --2- 172.21.15.27:0/234863999 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6760066f30 0x7f67600693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:39.018 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.016+0000 7f675b7fe700 1 -- 172.21.15.27:0/234863999 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5292+0+0 (secure 0 0 0) 0x7f67640859b0 con 0x7f67740b3940 2023-10-06T01:42:39.019 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.016+0000 7f6772ffd700 1 --2- 172.21.15.27:0/234863999 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6760066f30 0x7f67600693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:39.019 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.016+0000 7f675b7fe700 1 --2- 172.21.15.27:0/234863999 >> [v2:172.21.15.124:6808/90864,v1:172.21.15.124:6809/90864] conn(0x7f676006d3a0 0x7f676006f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:39.019 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.016+0000 7f675b7fe700 1 -- 172.21.15.27:0/234863999 --> [v2:172.21.15.124:6808/90864,v1:172.21.15.124:6809/90864] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f676006fe80 con 0x7f676006d3a0 2023-10-06T01:42:39.019 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.016+0000 7f675b7fe700 1 -- 172.21.15.27:0/234863999 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f676408a240 con 0x7f67740b3940 2023-10-06T01:42:39.019 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.016+0000 7f6773fff700 1 --2- 172.21.15.27:0/234863999 >> [v2:172.21.15.124:6808/90864,v1:172.21.15.124:6809/90864] conn(0x7f676006d3a0 0x7f676006f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:39.019 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.016+0000 7f6772ffd700 1 --2- 172.21.15.27:0/234863999 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6760066f30 0x7f67600693e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f67740c7bf0 tx=0x7f6768009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:39.019 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.016+0000 7f6773fff700 1 --2- 172.21.15.27:0/234863999 >> [v2:172.21.15.124:6808/90864,v1:172.21.15.124:6809/90864] conn(0x7f676006d3a0 0x7f676006f7b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:39.020 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.017+0000 7f675b7fe700 1 -- 172.21.15.27:0/234863999 <== osd.6 v2:172.21.15.124:6808/90864 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f676006fe80 con 0x7f676006d3a0 2023-10-06T01:42:39.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.031+0000 7f6779d4c700 1 -- 172.21.15.27:0/234863999 --> [v2:172.21.15.124:6808/90864,v1:172.21.15.124:6809/90864] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f677406a2b0 con 0x7f676006d3a0 2023-10-06T01:42:39.031 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.032+0000 7f675b7fe700 1 -- 172.21.15.27:0/234863999 <== osd.6 v2:172.21.15.124:6808/90864 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f677406a2b0 con 0x7f676006d3a0 2023-10-06T01:42:39.032 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2023-10-06T01:42:39.032 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-06T01:42:39.032 INFO:teuthology.orchestra.run.smithi027.stdout:} 2023-10-06T01:42:39.033 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.032+0000 7f6779d4c700 1 -- 172.21.15.27:0/234863999 >> [v2:172.21.15.124:6808/90864,v1:172.21.15.124:6809/90864] conn(0x7f676006d3a0 msgr2=0x7f676006f7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:39.033 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.032+0000 7f6779d4c700 1 --2- 172.21.15.27:0/234863999 >> [v2:172.21.15.124:6808/90864,v1:172.21.15.124:6809/90864] conn(0x7f676006d3a0 0x7f676006f7b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:39.033 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.032+0000 7f6779d4c700 1 -- 172.21.15.27:0/234863999 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6760066f30 msgr2=0x7f67600693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:39.033 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.032+0000 7f6779d4c700 1 --2- 172.21.15.27:0/234863999 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6760066f30 0x7f67600693e0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f67740c7bf0 tx=0x7f6768009040 comp rx=0 tx=0).stop 2023-10-06T01:42:39.033 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.032+0000 7f6779d4c700 1 -- 172.21.15.27:0/234863999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67740b3940 msgr2=0x7f6774147000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:39.033 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.032+0000 7f6779d4c700 1 --2- 172.21.15.27:0/234863999 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f67740b3940 0x7f6774147000 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f6764009f80 tx=0x7f6764017000 comp rx=0 tx=0).stop 2023-10-06T01:42:39.033 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.032+0000 7f6773fff700 1 -- 172.21.15.27:0/234863999 reap_dead start 2023-10-06T01:42:39.034 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.032+0000 7f6779d4c700 1 -- 172.21.15.27:0/234863999 shutdown_connections 2023-10-06T01:42:39.034 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.032+0000 7f6779d4c700 1 -- 172.21.15.27:0/234863999 >> 172.21.15.27:0/234863999 conn(0x7f67740ce4c0 msgr2=0x7f677415ca50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:39.034 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.032+0000 7f6779d4c700 1 -- 172.21.15.27:0/234863999 shutdown_connections 2023-10-06T01:42:39.034 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.032+0000 7f6779d4c700 1 -- 172.21.15.27:0/234863999 wait complete. 2023-10-06T01:42:39.044 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-10-06T01:42:39.163 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.163+0000 7f6fd759e700 1 Processor -- start 2023-10-06T01:42:39.164 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.163+0000 7f6fd759e700 1 -- start start 2023-10-06T01:42:39.166 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.164+0000 7f6fd759e700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6fd81593f0 0x7f6fd80b00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:39.166 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.164+0000 7f6fd759e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6fd81597c0 0x7f6fd80cf680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:39.166 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.164+0000 7f6fd759e700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd80cab50 con 0x7f6fd80cfbc0 2023-10-06T01:42:39.166 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.164+0000 7f6fd759e700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fd80cb3f0 con 0x7f6fd81593f0 2023-10-06T01:42:39.166 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.164+0000 7f6fd759e700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd80b05f0 con 0x7f6fd81597c0 2023-10-06T01:42:39.166 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.164+0000 7f6fd659c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6fd81593f0 0x7f6fd80b00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:39.166 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.164+0000 7f6fd659c700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f6fd81593f0 0x7f6fd80b00b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:45996/0 (socket says 172.21.15.27:45996) 2023-10-06T01:42:39.166 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.164+0000 7f6fd659c700 1 -- 172.21.15.27:0/305653174 learned_addr learned my addr 172.21.15.27:0/305653174 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:39.167 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.164+0000 7f6fd5d9b700 1 --2- 172.21.15.27:0/305653174 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6fd81597c0 0x7f6fd80cf680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:39.167 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.164+0000 7f6fd659c700 1 -- 172.21.15.27:0/305653174 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6fd81597c0 msgr2=0x7f6fd80cf680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:39.167 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.164+0000 7f6fd659c700 1 --2- 172.21.15.27:0/305653174 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6fd81597c0 0x7f6fd80cf680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:39.167 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.164+0000 7f6fd659c700 1 -- 172.21.15.27:0/305653174 >> v1:172.21.15.124:6789/0 conn(0x7f6fd80cfbc0 legacy=0x7f6fd8068350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:39.167 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.164+0000 7f6fd659c700 1 -- 172.21.15.27:0/305653174 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd80d0150 con 0x7f6fd81593f0 2023-10-06T01:42:39.167 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.164+0000 7f6fd659c700 1 --2- 172.21.15.27:0/305653174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fd81593f0 0x7f6fd80b00b0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f6fd00099b0 tx=0x7f6fd0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69602d7aad16133 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:39.167 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.165+0000 7f6fd559a700 1 -- 172.21.15.27:0/305653174 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd000b860 con 0x7f6fd81593f0 2023-10-06T01:42:39.167 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.165+0000 7f6fd559a700 1 -- 172.21.15.27:0/305653174 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd000b9c0 con 0x7f6fd81593f0 2023-10-06T01:42:39.167 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.165+0000 7f6fd559a700 1 -- 172.21.15.27:0/305653174 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd000bd60 con 0x7f6fd81593f0 2023-10-06T01:42:39.167 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.165+0000 7f6fd759e700 1 -- 172.21.15.27:0/305653174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fd81593f0 msgr2=0x7f6fd80b00b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:39.167 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.165+0000 7f6fd759e700 1 --2- 172.21.15.27:0/305653174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fd81593f0 0x7f6fd80b00b0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f6fd00099b0 tx=0x7f6fd0009250 comp rx=0 tx=0).stop 2023-10-06T01:42:39.167 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.165+0000 7f6fd759e700 1 -- 172.21.15.27:0/305653174 shutdown_connections 2023-10-06T01:42:39.168 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.165+0000 7f6fd759e700 1 --2- 172.21.15.27:0/305653174 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fd81593f0 0x7f6fd80b00b0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:39.168 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.165+0000 7f6fd759e700 1 --2- 172.21.15.27:0/305653174 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6fd81597c0 0x7f6fd80cf680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:39.168 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.165+0000 7f6fd759e700 1 -- 172.21.15.27:0/305653174 >> 172.21.15.27:0/305653174 conn(0x7f6fd80d0510 msgr2=0x7f6fd80d0910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:39.168 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.165+0000 7f6fd759e700 1 -- 172.21.15.27:0/305653174 shutdown_connections 2023-10-06T01:42:39.168 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.165+0000 7f6fd759e700 1 -- 172.21.15.27:0/305653174 wait complete. 2023-10-06T01:42:39.170 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.166+0000 7f6fd759e700 1 Processor -- start 2023-10-06T01:42:39.170 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.166+0000 7f6fd759e700 1 -- start start 2023-10-06T01:42:39.170 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.167+0000 7f6fd759e700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fd80cfbc0 0x7f6fd81530d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:39.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.167+0000 7f6fd759e700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6fd81593f0 0x7f6fd815e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:39.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.167+0000 7f6fd759e700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6fd81597c0 0x7f6fd80b1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:39.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.167+0000 7f6fd759e700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd815dd60 con 0x7f6fd80cfbc0 2023-10-06T01:42:39.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.167+0000 7f6fd759e700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd8153610 con 0x7f6fd81597c0 2023-10-06T01:42:39.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.167+0000 7f6fd759e700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd815d150 con 0x7f6fd81593f0 2023-10-06T01:42:39.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.167+0000 7f6fd659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fd80cfbc0 0x7f6fd81530d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:39.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.167+0000 7f6fd659c700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fd80cfbc0 0x7f6fd81530d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46012/0 (socket says 172.21.15.27:46012) 2023-10-06T01:42:39.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.167+0000 7f6fd659c700 1 -- 172.21.15.27:0/3563748575 learned_addr learned my addr 172.21.15.27:0/3563748575 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:39.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.167+0000 7f6fd5d9b700 1 --2- 172.21.15.27:0/3563748575 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6fd81593f0 0x7f6fd815e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:39.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.167+0000 7f6fd6d9d700 1 --2- 172.21.15.27:0/3563748575 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6fd81597c0 0x7f6fd80b1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:39.171 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.167+0000 7f6fd659c700 1 -- 172.21.15.27:0/3563748575 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6fd81593f0 msgr2=0x7f6fd815e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:39.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.167+0000 7f6fd659c700 1 --2- 172.21.15.27:0/3563748575 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6fd81593f0 0x7f6fd815e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:39.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.167+0000 7f6fd659c700 1 -- 172.21.15.27:0/3563748575 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6fd81597c0 msgr2=0x7f6fd80b1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:39.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.167+0000 7f6fd659c700 1 --2- 172.21.15.27:0/3563748575 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6fd81597c0 0x7f6fd80b1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:39.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.168+0000 7f6fd659c700 1 -- 172.21.15.27:0/3563748575 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd00094f0 con 0x7f6fd80cfbc0 2023-10-06T01:42:39.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.168+0000 7f6fd659c700 1 --2- 172.21.15.27:0/3563748575 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fd80cfbc0 0x7f6fd81530d0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f6fd80cc6f0 tx=0x7f6fd0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:39.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.168+0000 7f6fce7fc700 1 -- 172.21.15.27:0/3563748575 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd001a420 con 0x7f6fd80cfbc0 2023-10-06T01:42:39.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.168+0000 7f6fce7fc700 1 -- 172.21.15.27:0/3563748575 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd001a580 con 0x7f6fd80cfbc0 2023-10-06T01:42:39.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.168+0000 7f6fd759e700 1 -- 172.21.15.27:0/3563748575 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fd815d430 con 0x7f6fd80cfbc0 2023-10-06T01:42:39.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.168+0000 7f6fce7fc700 1 -- 172.21.15.27:0/3563748575 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd001a940 con 0x7f6fd80cfbc0 2023-10-06T01:42:39.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.168+0000 7f6fd759e700 1 -- 172.21.15.27:0/3563748575 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fd815d9f0 con 0x7f6fd80cfbc0 2023-10-06T01:42:39.172 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.169+0000 7f6fd759e700 1 -- 172.21.15.27:0/3563748575 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd804c3c0 con 0x7f6fd80cfbc0 2023-10-06T01:42:39.173 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.169+0000 7f6fce7fc700 1 -- 172.21.15.27:0/3563748575 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f6fd001acf0 con 0x7f6fd80cfbc0 2023-10-06T01:42:39.173 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.170+0000 7f6fce7fc700 1 --2- 172.21.15.27:0/3563748575 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6fb8066f30 0x7f6fb80693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:39.173 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.170+0000 7f6fce7fc700 1 -- 172.21.15.27:0/3563748575 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5292+0+0 (secure 0 0 0) 0x7f6fd0085d90 con 0x7f6fd80cfbc0 2023-10-06T01:42:39.173 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.170+0000 7f6fd5d9b700 1 --2- 172.21.15.27:0/3563748575 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6fb8066f30 0x7f6fb80693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:39.173 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.170+0000 7f6fd5d9b700 1 --2- 172.21.15.27:0/3563748575 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6fb8066f30 0x7f6fb80693e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6fd80af190 tx=0x7f6fc0008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:39.174 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.174+0000 7f6fce7fc700 1 -- 172.21.15.27:0/3563748575 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6fd00531a0 con 0x7f6fd80cfbc0 2023-10-06T01:42:39.366 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.366+0000 7f6fd759e700 1 -- 172.21.15.27:0/3563748575 --> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f6fd815dea0 con 0x7f6fb8066f30 2023-10-06T01:42:39.367 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.367+0000 7f6fce7fc700 1 -- 172.21.15.27:0/3563748575 <== mgr.4100 v2:172.21.15.124:6832/90278 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6fc000f6e0 con 0x7f6fb8066f30 2023-10-06T01:42:39.369 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.369+0000 7f6fd759e700 1 -- 172.21.15.27:0/3563748575 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6fb8066f30 msgr2=0x7f6fb80693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:39.370 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.369+0000 7f6fd759e700 1 --2- 172.21.15.27:0/3563748575 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6fb8066f30 0x7f6fb80693e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6fd80af190 tx=0x7f6fc0008040 comp rx=0 tx=0).stop 2023-10-06T01:42:39.370 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.369+0000 7f6fd759e700 1 -- 172.21.15.27:0/3563748575 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fd80cfbc0 msgr2=0x7f6fd81530d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:39.370 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.369+0000 7f6fd759e700 1 --2- 172.21.15.27:0/3563748575 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fd80cfbc0 0x7f6fd81530d0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f6fd80cc6f0 tx=0x7f6fd0017000 comp rx=0 tx=0).stop 2023-10-06T01:42:39.370 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.369+0000 7f6fd759e700 1 -- 172.21.15.27:0/3563748575 shutdown_connections 2023-10-06T01:42:39.370 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.369+0000 7f6fd759e700 1 --2- 172.21.15.27:0/3563748575 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f6fb8066f30 0x7f6fb80693e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:39.370 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.369+0000 7f6fd759e700 1 --2- 172.21.15.27:0/3563748575 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f6fd81597c0 0x7f6fd80b1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:39.370 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.369+0000 7f6fd759e700 1 --2- 172.21.15.27:0/3563748575 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f6fd80cfbc0 0x7f6fd81530d0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:39.370 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.369+0000 7f6fd759e700 1 --2- 172.21.15.27:0/3563748575 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f6fd81593f0 0x7f6fd815e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:39.370 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.369+0000 7f6fd759e700 1 -- 172.21.15.27:0/3563748575 >> 172.21.15.27:0/3563748575 conn(0x7f6fd80d0510 msgr2=0x7f6fd8154da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:39.370 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.369+0000 7f6fd759e700 1 -- 172.21.15.27:0/3563748575 shutdown_connections 2023-10-06T01:42:39.371 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.369+0000 7f6fd759e700 1 -- 172.21.15.27:0/3563748575 wait complete. 2023-10-06T01:42:39.371 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 6 to deep-scrub 2023-10-06T01:42:39.381 INFO:tasks.ceph:Scrubbing osd.7 2023-10-06T01:42:39.381 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-10-06T01:42:39.504 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.504+0000 7f298d293700 1 Processor -- start 2023-10-06T01:42:39.505 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.504+0000 7f298d293700 1 -- start start 2023-10-06T01:42:39.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.505+0000 7f298d293700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f29880cf440 0x7f29880cbb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:39.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.505+0000 7f298d293700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29880cb670 con 0x7f29880cc850 2023-10-06T01:42:39.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.505+0000 7f298d293700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29880cce90 con 0x7f29880cf810 2023-10-06T01:42:39.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.505+0000 7f298d293700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29880cccb0 con 0x7f29880cf440 2023-10-06T01:42:39.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.505+0000 7f2987fff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f29880cf440 0x7f29880cbb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:39.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.505+0000 7f298ca92700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f29880cc850 0x7f2988158e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47398/0 (socket says 172.21.15.27:47398) 2023-10-06T01:42:39.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.505+0000 7f298ca92700 1 -- 172.21.15.27:0/3447654530 learned_addr learned my addr 172.21.15.27:0/3447654530 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:39.506 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.505+0000 7f2987fff700 1 -- 172.21.15.27:0/3447654530 >> v1:172.21.15.124:6789/0 conn(0x7f29880cf810 legacy=0x7f29880cc140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:39.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.506+0000 7f2987fff700 1 -- 172.21.15.27:0/3447654530 >> v1:172.21.15.27:6789/0 conn(0x7f29880cc850 legacy=0x7f2988158e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:39.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.506+0000 7f2987fff700 1 -- 172.21.15.27:0/3447654530 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29880cad80 con 0x7f29880cf440 2023-10-06T01:42:39.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.506+0000 7f2987fff700 1 --2- 172.21.15.27:0/3447654530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29880cf440 0x7f29880cbb00 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f29700098e0 tx=0x7f29700091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c50998c05df3c219 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:39.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.506+0000 7f2986ffd700 1 -- 172.21.15.27:0/3447654530 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2970009e80 con 0x7f29880cf440 2023-10-06T01:42:39.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.506+0000 7f2986ffd700 1 -- 172.21.15.27:0/3447654530 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f297000b820 con 0x7f29880cf440 2023-10-06T01:42:39.507 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.506+0000 7f2986ffd700 1 -- 172.21.15.27:0/3447654530 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f297000bba0 con 0x7f29880cf440 2023-10-06T01:42:39.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.506+0000 7f298d293700 1 -- 172.21.15.27:0/3447654530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29880cf440 msgr2=0x7f29880cbb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:39.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.506+0000 7f298d293700 1 --2- 172.21.15.27:0/3447654530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29880cf440 0x7f29880cbb00 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f29700098e0 tx=0x7f29700091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:39.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.506+0000 7f298d293700 1 -- 172.21.15.27:0/3447654530 shutdown_connections 2023-10-06T01:42:39.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.506+0000 7f298d293700 1 --2- 172.21.15.27:0/3447654530 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29880cf440 0x7f29880cbb00 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:39.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.506+0000 7f298d293700 1 -- 172.21.15.27:0/3447654530 >> 172.21.15.27:0/3447654530 conn(0x7f29880cd080 msgr2=0x7f29880cd480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:39.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.506+0000 7f298d293700 1 -- 172.21.15.27:0/3447654530 shutdown_connections 2023-10-06T01:42:39.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.506+0000 7f298d293700 1 -- 172.21.15.27:0/3447654530 wait complete. 2023-10-06T01:42:39.508 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.507+0000 7f298d293700 1 Processor -- start 2023-10-06T01:42:39.509 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.508+0000 7f298d293700 1 -- start start 2023-10-06T01:42:39.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.508+0000 7f298d293700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29880cc850 0x7f298816d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:39.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.508+0000 7f298d293700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f29880cf440 0x7f298815e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:39.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.508+0000 7f298d293700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f29880cf810 0x7f2988169bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:39.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.508+0000 7f298d293700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298815e8e0 con 0x7f29880cc850 2023-10-06T01:42:39.511 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.508+0000 7f298d293700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29880b4b60 con 0x7f29880cf440 2023-10-06T01:42:39.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.508+0000 7f298d293700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f298816e820 con 0x7f29880cf810 2023-10-06T01:42:39.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.508+0000 7f2987fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29880cc850 0x7f298816d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:39.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.508+0000 7f2987fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29880cc850 0x7f298816d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46022/0 (socket says 172.21.15.27:46022) 2023-10-06T01:42:39.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.508+0000 7f2987fff700 1 -- 172.21.15.27:0/862778675 learned_addr learned my addr 172.21.15.27:0/862778675 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:39.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.509+0000 7f298ca92700 1 --2- 172.21.15.27:0/862778675 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f29880cf810 0x7f2988169bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:39.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.509+0000 7f29877fe700 1 --2- 172.21.15.27:0/862778675 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f29880cf440 0x7f298815e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:39.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.509+0000 7f2987fff700 1 -- 172.21.15.27:0/862778675 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f29880cf810 msgr2=0x7f2988169bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:39.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.509+0000 7f2987fff700 1 --2- 172.21.15.27:0/862778675 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f29880cf810 0x7f2988169bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:39.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.509+0000 7f2987fff700 1 -- 172.21.15.27:0/862778675 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f29880cf440 msgr2=0x7f298815e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:39.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.509+0000 7f2987fff700 1 --2- 172.21.15.27:0/862778675 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f29880cf440 0x7f298815e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:39.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.509+0000 7f2987fff700 1 -- 172.21.15.27:0/862778675 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2970009420 con 0x7f29880cc850 2023-10-06T01:42:39.512 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.509+0000 7f2987fff700 1 --2- 172.21.15.27:0/862778675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29880cc850 0x7f298816d8f0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f2988089360 tx=0x7f2970016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:39.513 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.509+0000 7f297ffff700 1 -- 172.21.15.27:0/862778675 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f297000bae0 con 0x7f29880cc850 2023-10-06T01:42:39.513 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.509+0000 7f297ffff700 1 -- 172.21.15.27:0/862778675 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2970020070 con 0x7f29880cc850 2023-10-06T01:42:39.513 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.509+0000 7f297ffff700 1 -- 172.21.15.27:0/862778675 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f297001b6a0 con 0x7f29880cc850 2023-10-06T01:42:39.513 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.509+0000 7f298d293700 1 -- 172.21.15.27:0/862778675 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f298816df80 con 0x7f29880cc850 2023-10-06T01:42:39.513 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.509+0000 7f298d293700 1 -- 172.21.15.27:0/862778675 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f298816e540 con 0x7f29880cc850 2023-10-06T01:42:39.513 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.510+0000 7f298d293700 1 -- 172.21.15.27:0/862778675 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2988069490 con 0x7f29880cc850 2023-10-06T01:42:39.513 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.510+0000 7f297ffff700 1 -- 172.21.15.27:0/862778675 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2970022030 con 0x7f29880cc850 2023-10-06T01:42:39.513 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.511+0000 7f297ffff700 1 --2- 172.21.15.27:0/862778675 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2968066f30 0x7f29680693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:39.513 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.511+0000 7f297ffff700 1 -- 172.21.15.27:0/862778675 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5292+0+0 (secure 0 0 0) 0x7f2970086330 con 0x7f29880cc850 2023-10-06T01:42:39.513 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.511+0000 7f297ffff700 1 --2- 172.21.15.27:0/862778675 >> [v2:172.21.15.124:6824/90894,v1:172.21.15.124:6825/90894] conn(0x7f296806d3a0 0x7f296806f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:39.513 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.511+0000 7f297ffff700 1 -- 172.21.15.27:0/862778675 --> [v2:172.21.15.124:6824/90894,v1:172.21.15.124:6825/90894] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f296806fe80 con 0x7f296806d3a0 2023-10-06T01:42:39.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.511+0000 7f29877fe700 1 --2- 172.21.15.27:0/862778675 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2968066f30 0x7f29680693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:39.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.511+0000 7f297ffff700 1 -- 172.21.15.27:0/862778675 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2970088910 con 0x7f29880cc850 2023-10-06T01:42:39.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.511+0000 7f298ca92700 1 --2- 172.21.15.27:0/862778675 >> [v2:172.21.15.124:6824/90894,v1:172.21.15.124:6825/90894] conn(0x7f296806d3a0 0x7f296806f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:39.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.511+0000 7f29877fe700 1 --2- 172.21.15.27:0/862778675 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2968066f30 0x7f29680693e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2978006fd0 tx=0x7f2978008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:39.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.512+0000 7f298ca92700 1 --2- 172.21.15.27:0/862778675 >> [v2:172.21.15.124:6824/90894,v1:172.21.15.124:6825/90894] conn(0x7f296806d3a0 0x7f296806f7b0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:39.514 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.512+0000 7f297ffff700 1 -- 172.21.15.27:0/862778675 <== osd.7 v2:172.21.15.124:6824/90894 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f296806fe80 con 0x7f296806d3a0 2023-10-06T01:42:39.526 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.526+0000 7f298d293700 1 -- 172.21.15.27:0/862778675 --> [v2:172.21.15.124:6824/90894,v1:172.21.15.124:6825/90894] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f2988064050 con 0x7f296806d3a0 2023-10-06T01:42:39.527 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.527+0000 7f297ffff700 1 -- 172.21.15.27:0/862778675 <== osd.7 v2:172.21.15.124:6824/90894 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f2988064050 con 0x7f296806d3a0 2023-10-06T01:42:39.527 INFO:teuthology.orchestra.run.smithi027.stdout:{ 2023-10-06T01:42:39.527 INFO:teuthology.orchestra.run.smithi027.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-06T01:42:39.527 INFO:teuthology.orchestra.run.smithi027.stdout:} 2023-10-06T01:42:39.528 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.527+0000 7f298d293700 1 -- 172.21.15.27:0/862778675 >> [v2:172.21.15.124:6824/90894,v1:172.21.15.124:6825/90894] conn(0x7f296806d3a0 msgr2=0x7f296806f7b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:39.529 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.527+0000 7f298d293700 1 --2- 172.21.15.27:0/862778675 >> [v2:172.21.15.124:6824/90894,v1:172.21.15.124:6825/90894] conn(0x7f296806d3a0 0x7f296806f7b0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:39.529 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.527+0000 7f298d293700 1 -- 172.21.15.27:0/862778675 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2968066f30 msgr2=0x7f29680693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:39.529 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.527+0000 7f298d293700 1 --2- 172.21.15.27:0/862778675 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2968066f30 0x7f29680693e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f2978006fd0 tx=0x7f2978008040 comp rx=0 tx=0).stop 2023-10-06T01:42:39.529 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.527+0000 7f298d293700 1 -- 172.21.15.27:0/862778675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29880cc850 msgr2=0x7f298816d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:39.529 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.527+0000 7f298d293700 1 --2- 172.21.15.27:0/862778675 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f29880cc850 0x7f298816d8f0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f2988089360 tx=0x7f2970016000 comp rx=0 tx=0).stop 2023-10-06T01:42:39.529 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.527+0000 7f298ca92700 1 -- 172.21.15.27:0/862778675 reap_dead start 2023-10-06T01:42:39.529 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.527+0000 7f298d293700 1 -- 172.21.15.27:0/862778675 shutdown_connections 2023-10-06T01:42:39.529 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.527+0000 7f298d293700 1 -- 172.21.15.27:0/862778675 >> 172.21.15.27:0/862778675 conn(0x7f29880cd080 msgr2=0x7f29880b27f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:39.529 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.527+0000 7f298d293700 1 -- 172.21.15.27:0/862778675 shutdown_connections 2023-10-06T01:42:39.529 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.527+0000 7f298d293700 1 -- 172.21.15.27:0/862778675 wait complete. 2023-10-06T01:42:39.539 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-10-06T01:42:39.659 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.658+0000 7f7df94ff700 1 Processor -- start 2023-10-06T01:42:39.659 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.658+0000 7f7df94ff700 1 -- start start 2023-10-06T01:42:39.661 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.659+0000 7f7df94ff700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7df4159500 0x7f7df41598d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:39.661 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.659+0000 7f7df94ff700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7df40b3fd0 0x7f7df415d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:39.662 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.659+0000 7f7df94ff700 1 -- --> v1:172.21.15.27:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7df415dd20 con 0x7f7df415d8e0 2023-10-06T01:42:39.662 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.659+0000 7f7df94ff700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7df40b1110 con 0x7f7df40b3fd0 2023-10-06T01:42:39.662 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.659+0000 7f7df94ff700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7df40b13b0 con 0x7f7df4159500 2023-10-06T01:42:39.662 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.659+0000 7f7df27fc700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f7df40b3fd0 0x7f7df415d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:39.662 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.659+0000 7f7df37fe700 1 --1- >> v1:172.21.15.27:6789/0 conn(0x7f7df415d8e0 0x7f7df40cf6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.27:6789/0 says I am v1:172.21.15.27:47410/0 (socket says 172.21.15.27:47410) 2023-10-06T01:42:39.662 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.659+0000 7f7df37fe700 1 -- 172.21.15.27:0/3007189419 learned_addr learned my addr 172.21.15.27:0/3007189419 (peer_addr_for_me v1:172.21.15.27:0/0) 2023-10-06T01:42:39.662 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.659+0000 7f7df2ffd700 1 --2- 172.21.15.27:0/3007189419 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7df4159500 0x7f7df41598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:39.663 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.660+0000 7f7df27fc700 1 -- 172.21.15.27:0/3007189419 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7df4159500 msgr2=0x7f7df41598d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:39.663 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.660+0000 7f7df27fc700 1 --2- 172.21.15.27:0/3007189419 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7df4159500 0x7f7df41598d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:39.663 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.660+0000 7f7df27fc700 1 -- 172.21.15.27:0/3007189419 >> v1:172.21.15.27:6789/0 conn(0x7f7df415d8e0 legacy=0x7f7df40cf6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:39.663 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.660+0000 7f7df27fc700 1 -- 172.21.15.27:0/3007189419 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df40b1690 con 0x7f7df40b3fd0 2023-10-06T01:42:39.663 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.660+0000 7f7df1ffb700 1 -- 172.21.15.27:0/3007189419 <== mon.0 v1:172.21.15.27:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1740212260 0 0) 0x7f7df415dd20 con 0x7f7df415d8e0 2023-10-06T01:42:39.663 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.660+0000 7f7df27fc700 1 --2- 172.21.15.27:0/3007189419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7df40b3fd0 0x7f7df415d3a0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc0098e0 tx=0x7f7ddc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=365d97d09bbe4ed5 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:39.664 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.660+0000 7f7df1ffb700 1 -- 172.21.15.27:0/3007189419 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc00b640 con 0x7f7df40b3fd0 2023-10-06T01:42:39.664 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.660+0000 7f7df1ffb700 1 -- 172.21.15.27:0/3007189419 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ddc00b7a0 con 0x7f7df40b3fd0 2023-10-06T01:42:39.664 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.660+0000 7f7df1ffb700 1 -- 172.21.15.27:0/3007189419 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc00bb40 con 0x7f7df40b3fd0 2023-10-06T01:42:39.664 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.660+0000 7f7df94ff700 1 -- 172.21.15.27:0/3007189419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7df40b3fd0 msgr2=0x7f7df415d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:39.664 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.660+0000 7f7df94ff700 1 --2- 172.21.15.27:0/3007189419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7df40b3fd0 0x7f7df415d3a0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc0098e0 tx=0x7f7ddc0091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:39.665 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.661+0000 7f7df94ff700 1 -- 172.21.15.27:0/3007189419 shutdown_connections 2023-10-06T01:42:39.665 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.661+0000 7f7df94ff700 1 --2- 172.21.15.27:0/3007189419 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7df4159500 0x7f7df41598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:39.665 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.661+0000 7f7df94ff700 1 --2- 172.21.15.27:0/3007189419 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7df40b3fd0 0x7f7df415d3a0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:39.665 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.661+0000 7f7df94ff700 1 -- 172.21.15.27:0/3007189419 >> 172.21.15.27:0/3007189419 conn(0x7f7df4067630 msgr2=0x7f7df4069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:39.665 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.661+0000 7f7df94ff700 1 -- 172.21.15.27:0/3007189419 shutdown_connections 2023-10-06T01:42:39.666 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.661+0000 7f7df94ff700 1 -- 172.21.15.27:0/3007189419 wait complete. 2023-10-06T01:42:39.669 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.662+0000 7f7df94ff700 1 Processor -- start 2023-10-06T01:42:39.669 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.662+0000 7f7df94ff700 1 -- start start 2023-10-06T01:42:39.669 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.662+0000 7f7df94ff700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7df40b3fd0 0x7f7df416dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:39.669 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.662+0000 7f7df94ff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7df4159500 0x7f7df414b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:39.669 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.663+0000 7f7df94ff700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7df415d8e0 0x7f7df40cee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:39.669 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.663+0000 7f7df94ff700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df414bb90 con 0x7f7df4159500 2023-10-06T01:42:39.669 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.663+0000 7f7df94ff700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df415a630 con 0x7f7df415d8e0 2023-10-06T01:42:39.670 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.663+0000 7f7df94ff700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7df416e1b0 con 0x7f7df40b3fd0 2023-10-06T01:42:39.670 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.663+0000 7f7df27fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7df4159500 0x7f7df414b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:39.670 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.663+0000 7f7df27fc700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7df4159500 0x7f7df414b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46048/0 (socket says 172.21.15.27:46048) 2023-10-06T01:42:39.670 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.663+0000 7f7df27fc700 1 -- 172.21.15.27:0/620486528 learned_addr learned my addr 172.21.15.27:0/620486528 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:39.670 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.663+0000 7f7df2ffd700 1 --2- 172.21.15.27:0/620486528 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7df40b3fd0 0x7f7df416dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:39.670 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.663+0000 7f7df37fe700 1 --2- 172.21.15.27:0/620486528 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7df415d8e0 0x7f7df40cee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:39.670 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.663+0000 7f7df27fc700 1 -- 172.21.15.27:0/620486528 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7df40b3fd0 msgr2=0x7f7df416dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:39.670 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.663+0000 7f7df27fc700 1 --2- 172.21.15.27:0/620486528 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7df40b3fd0 0x7f7df416dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:39.670 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.663+0000 7f7df27fc700 1 -- 172.21.15.27:0/620486528 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7df415d8e0 msgr2=0x7f7df40cee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:39.671 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.663+0000 7f7df27fc700 1 --2- 172.21.15.27:0/620486528 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7df415d8e0 0x7f7df40cee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:39.671 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.663+0000 7f7df27fc700 1 -- 172.21.15.27:0/620486528 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ddc009420 con 0x7f7df4159500 2023-10-06T01:42:39.671 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.663+0000 7f7df37fe700 1 --2- 172.21.15.27:0/620486528 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7df415d8e0 0x7f7df40cee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:42:39.671 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.663+0000 7f7df27fc700 1 --2- 172.21.15.27:0/620486528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7df4159500 0x7f7df414b050 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc009f80 tx=0x7f7ddc017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:39.671 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.663+0000 7f7deaffd700 1 -- 172.21.15.27:0/620486528 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc00bb20 con 0x7f7df4159500 2023-10-06T01:42:39.671 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.663+0000 7f7deaffd700 1 -- 172.21.15.27:0/620486528 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ddc000c90 con 0x7f7df4159500 2023-10-06T01:42:39.671 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.663+0000 7f7df94ff700 1 -- 172.21.15.27:0/620486528 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7df414b590 con 0x7f7df4159500 2023-10-06T01:42:39.671 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.663+0000 7f7deaffd700 1 -- 172.21.15.27:0/620486528 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc011630 con 0x7f7df4159500 2023-10-06T01:42:39.671 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.663+0000 7f7df94ff700 1 -- 172.21.15.27:0/620486528 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7df414d570 con 0x7f7df4159500 2023-10-06T01:42:39.672 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.664+0000 7f7df94ff700 1 -- 172.21.15.27:0/620486528 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7df4064050 con 0x7f7df4159500 2023-10-06T01:42:39.672 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.665+0000 7f7deaffd700 1 -- 172.21.15.27:0/620486528 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f7ddc0119a0 con 0x7f7df4159500 2023-10-06T01:42:39.672 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.665+0000 7f7deaffd700 1 --2- 172.21.15.27:0/620486528 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7dd4066f30 0x7f7dd40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:39.672 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.665+0000 7f7deaffd700 1 -- 172.21.15.27:0/620486528 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5292+0+0 (secure 0 0 0) 0x7f7ddc0856e0 con 0x7f7df4159500 2023-10-06T01:42:39.672 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.665+0000 7f7df2ffd700 1 --2- 172.21.15.27:0/620486528 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7dd4066f30 0x7f7dd40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:39.672 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.666+0000 7f7df2ffd700 1 --2- 172.21.15.27:0/620486528 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7dd4066f30 0x7f7dd40693e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7df416d210 tx=0x7f7dec0091a0 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:39.672 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.670+0000 7f7deaffd700 1 -- 172.21.15.27:0/620486528 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f7ddc052af0 con 0x7f7df4159500 2023-10-06T01:42:39.866 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.866+0000 7f7df94ff700 1 -- 172.21.15.27:0/620486528 --> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f7df40b1970 con 0x7f7dd4066f30 2023-10-06T01:42:39.866 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.866+0000 7f7deaffd700 1 -- 172.21.15.27:0/620486528 <== mgr.4100 v2:172.21.15.124:6832/90278 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7df40b1970 con 0x7f7dd4066f30 2023-10-06T01:42:39.869 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.868+0000 7f7df94ff700 1 -- 172.21.15.27:0/620486528 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7dd4066f30 msgr2=0x7f7dd40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:39.869 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.868+0000 7f7df94ff700 1 --2- 172.21.15.27:0/620486528 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7dd4066f30 0x7f7dd40693e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7df416d210 tx=0x7f7dec0091a0 comp rx=0 tx=0).stop 2023-10-06T01:42:39.869 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.868+0000 7f7df94ff700 1 -- 172.21.15.27:0/620486528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7df4159500 msgr2=0x7f7df414b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:39.869 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.869+0000 7f7df94ff700 1 --2- 172.21.15.27:0/620486528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7df4159500 0x7f7df414b050 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc009f80 tx=0x7f7ddc017000 comp rx=0 tx=0).stop 2023-10-06T01:42:39.870 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.869+0000 7f7df94ff700 1 -- 172.21.15.27:0/620486528 shutdown_connections 2023-10-06T01:42:39.870 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.869+0000 7f7df94ff700 1 --2- 172.21.15.27:0/620486528 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f7dd4066f30 0x7f7dd40693e0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:39.870 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.869+0000 7f7df94ff700 1 --2- 172.21.15.27:0/620486528 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f7df40b3fd0 0x7f7df416dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:39.870 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.869+0000 7f7df94ff700 1 --2- 172.21.15.27:0/620486528 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f7df415d8e0 0x7f7df40cee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:39.870 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.869+0000 7f7df94ff700 1 --2- 172.21.15.27:0/620486528 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f7df4159500 0x7f7df414b050 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:39.870 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.869+0000 7f7df94ff700 1 -- 172.21.15.27:0/620486528 >> 172.21.15.27:0/620486528 conn(0x7f7df4067630 msgr2=0x7f7df41583a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:39.870 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.869+0000 7f7df94ff700 1 -- 172.21.15.27:0/620486528 shutdown_connections 2023-10-06T01:42:39.870 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.869+0000 7f7df94ff700 1 -- 172.21.15.27:0/620486528 wait complete. 2023-10-06T01:42:39.870 INFO:teuthology.orchestra.run.smithi027.stderr:instructed osd(s) 7 to deep-scrub 2023-10-06T01:42:39.881 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-06T01:42:39.999 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.998+0000 7f2d3c55a700 1 Processor -- start 2023-10-06T01:42:39.999 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.999+0000 7f2d3c55a700 1 -- start start 2023-10-06T01:42:40.001 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.999+0000 7f2d3c55a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2d34142d90 0x7f2d3406a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:40.001 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.999+0000 7f2d3c55a700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2d34143160 0x7f2d340c43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:40.001 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.999+0000 7f2d3c55a700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d34069f50 con 0x7f2d340c4900 2023-10-06T01:42:40.001 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.999+0000 7f2d3c55a700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d3406a920 con 0x7f2d34142d90 2023-10-06T01:42:40.001 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:39.999+0000 7f2d3c55a700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d34069340 con 0x7f2d34143160 2023-10-06T01:42:40.002 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.000+0000 7f2d3a2f6700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2d34142d90 0x7f2d3406a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:40.002 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.000+0000 7f2d3a2f6700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f2d34142d90 0x7f2d3406a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46058/0 (socket says 172.21.15.27:46058) 2023-10-06T01:42:40.002 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.000+0000 7f2d3a2f6700 1 -- 172.21.15.27:0/1990612924 learned_addr learned my addr 172.21.15.27:0/1990612924 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:40.002 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.000+0000 7f2d39af5700 1 --2- 172.21.15.27:0/1990612924 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2d34143160 0x7f2d340c43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:40.002 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.000+0000 7f2d3a2f6700 1 -- 172.21.15.27:0/1990612924 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2d34143160 msgr2=0x7f2d340c43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:40.003 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.000+0000 7f2d3a2f6700 1 --2- 172.21.15.27:0/1990612924 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2d34143160 0x7f2d340c43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:40.003 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.000+0000 7f2d3a2f6700 1 -- 172.21.15.27:0/1990612924 >> v1:172.21.15.124:6789/0 conn(0x7f2d340c4900 legacy=0x7f2d340cc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:42:40.003 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.000+0000 7f2d3a2f6700 1 -- 172.21.15.27:0/1990612924 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d34069620 con 0x7f2d34142d90 2023-10-06T01:42:40.003 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.000+0000 7f2d3a2f6700 1 --2- 172.21.15.27:0/1990612924 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d34142d90 0x7f2d3406a3e0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c0099b0 tx=0x7f2d2c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d26b4c53bb3be55 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:40.003 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.000+0000 7f2d392f4700 1 -- 172.21.15.27:0/1990612924 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d2c00b820 con 0x7f2d34142d90 2023-10-06T01:42:40.003 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.000+0000 7f2d392f4700 1 -- 172.21.15.27:0/1990612924 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d2c00b980 con 0x7f2d34142d90 2023-10-06T01:42:40.003 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.000+0000 7f2d392f4700 1 -- 172.21.15.27:0/1990612924 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d2c00bd20 con 0x7f2d34142d90 2023-10-06T01:42:40.006 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.001+0000 7f2d3c55a700 1 -- 172.21.15.27:0/1990612924 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d34142d90 msgr2=0x7f2d3406a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:40.006 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.001+0000 7f2d3c55a700 1 --2- 172.21.15.27:0/1990612924 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d34142d90 0x7f2d3406a3e0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c0099b0 tx=0x7f2d2c009250 comp rx=0 tx=0).stop 2023-10-06T01:42:40.007 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.001+0000 7f2d3c55a700 1 -- 172.21.15.27:0/1990612924 shutdown_connections 2023-10-06T01:42:40.007 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.001+0000 7f2d3c55a700 1 --2- 172.21.15.27:0/1990612924 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d34142d90 0x7f2d3406a3e0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:40.007 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.001+0000 7f2d3c55a700 1 --2- 172.21.15.27:0/1990612924 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2d34143160 0x7f2d340c43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:40.007 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.001+0000 7f2d3c55a700 1 -- 172.21.15.27:0/1990612924 >> 172.21.15.27:0/1990612924 conn(0x7f2d340ce4d0 msgr2=0x7f2d340ce8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:40.007 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.001+0000 7f2d3c55a700 1 -- 172.21.15.27:0/1990612924 shutdown_connections 2023-10-06T01:42:40.007 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.001+0000 7f2d3c55a700 1 -- 172.21.15.27:0/1990612924 wait complete. 2023-10-06T01:42:40.008 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.002+0000 7f2d3c55a700 1 Processor -- start 2023-10-06T01:42:40.008 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.002+0000 7f2d3c55a700 1 -- start start 2023-10-06T01:42:40.008 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.003+0000 7f2d3c55a700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2d340c4900 0x7f2d3415b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:40.008 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.003+0000 7f2d3c55a700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2d34142d90 0x7f2d3413a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:40.009 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.003+0000 7f2d3c55a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d34143160 0x7f2d34159030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:40.009 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.003+0000 7f2d3c55a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d34138cb0 con 0x7f2d34143160 2023-10-06T01:42:40.009 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.003+0000 7f2d3c55a700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d34159570 con 0x7f2d340c4900 2023-10-06T01:42:40.009 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.003+0000 7f2d3c55a700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d3413b080 con 0x7f2d34142d90 2023-10-06T01:42:40.009 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.003+0000 7f2d3aaf7700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d34143160 0x7f2d34159030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:40.009 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.003+0000 7f2d3aaf7700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d34143160 0x7f2d34159030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:46062/0 (socket says 172.21.15.27:46062) 2023-10-06T01:42:40.009 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.003+0000 7f2d3aaf7700 1 -- 172.21.15.27:0/2633121364 learned_addr learned my addr 172.21.15.27:0/2633121364 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:42:40.010 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.003+0000 7f2d39af5700 1 --2- 172.21.15.27:0/2633121364 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2d34142d90 0x7f2d3413a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:40.010 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.003+0000 7f2d3a2f6700 1 --2- 172.21.15.27:0/2633121364 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2d340c4900 0x7f2d3415b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:40.010 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.003+0000 7f2d3aaf7700 1 -- 172.21.15.27:0/2633121364 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2d34142d90 msgr2=0x7f2d3413a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:40.010 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.003+0000 7f2d3aaf7700 1 --2- 172.21.15.27:0/2633121364 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2d34142d90 0x7f2d3413a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:40.010 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.003+0000 7f2d3aaf7700 1 -- 172.21.15.27:0/2633121364 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2d340c4900 msgr2=0x7f2d3415b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:40.011 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.003+0000 7f2d3aaf7700 1 --2- 172.21.15.27:0/2633121364 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2d340c4900 0x7f2d3415b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:40.011 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.003+0000 7f2d3aaf7700 1 -- 172.21.15.27:0/2633121364 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2c0094f0 con 0x7f2d34143160 2023-10-06T01:42:40.011 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.003+0000 7f2d3aaf7700 1 --2- 172.21.15.27:0/2633121364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d34143160 0x7f2d34159030 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f2d2400c890 tx=0x7f2d240026a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:40.011 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.004+0000 7f2d327fc700 1 -- 172.21.15.27:0/2633121364 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d24012630 con 0x7f2d34143160 2023-10-06T01:42:40.011 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.004+0000 7f2d327fc700 1 -- 172.21.15.27:0/2633121364 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d24012790 con 0x7f2d34143160 2023-10-06T01:42:40.011 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.004+0000 7f2d327fc700 1 -- 172.21.15.27:0/2633121364 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d24012b50 con 0x7f2d34143160 2023-10-06T01:42:40.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.004+0000 7f2d3c55a700 1 -- 172.21.15.27:0/2633121364 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d3413b360 con 0x7f2d34143160 2023-10-06T01:42:40.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.004+0000 7f2d3c55a700 1 -- 172.21.15.27:0/2633121364 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d34138940 con 0x7f2d34143160 2023-10-06T01:42:40.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.005+0000 7f2d3c55a700 1 -- 172.21.15.27:0/2633121364 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d34064050 con 0x7f2d34143160 2023-10-06T01:42:40.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.005+0000 7f2d327fc700 1 -- 172.21.15.27:0/2633121364 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f2d2400a420 con 0x7f2d34143160 2023-10-06T01:42:40.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.005+0000 7f2d327fc700 1 --2- 172.21.15.27:0/2633121364 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2d20066f30 0x7f2d200693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:42:40.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.005+0000 7f2d327fc700 1 -- 172.21.15.27:0/2633121364 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5292+0+0 (secure 0 0 0) 0x7f2d240847c0 con 0x7f2d34143160 2023-10-06T01:42:40.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.006+0000 7f2d3a2f6700 1 --2- 172.21.15.27:0/2633121364 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2d20066f30 0x7f2d200693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:42:40.012 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.006+0000 7f2d3a2f6700 1 --2- 172.21.15.27:0/2633121364 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2d20066f30 0x7f2d200693e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c0099b0 tx=0x7f2d2c016000 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:42:40.013 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.010+0000 7f2d327fc700 1 -- 172.21.15.27:0/2633121364 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2d24051bd0 con 0x7f2d34143160 2023-10-06T01:42:40.197 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.197+0000 7f2d3c55a700 1 -- 172.21.15.27:0/2633121364 --> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2d34138df0 con 0x7f2d20066f30 2023-10-06T01:42:40.200 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.200+0000 7f2d327fc700 1 -- 172.21.15.27:0/2633121364 <== mgr.4100 v2:172.21.15.124:6832/90278 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+73177 (secure 0 0 0) 0x7f2d34138df0 con 0x7f2d20066f30 2023-10-06T01:42:40.201 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:42:40.203 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.202+0000 7f2d3c55a700 1 -- 172.21.15.27:0/2633121364 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2d20066f30 msgr2=0x7f2d200693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:40.203 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.202+0000 7f2d3c55a700 1 --2- 172.21.15.27:0/2633121364 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2d20066f30 0x7f2d200693e0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2d2c0099b0 tx=0x7f2d2c016000 comp rx=0 tx=0).stop 2023-10-06T01:42:40.203 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.202+0000 7f2d3c55a700 1 -- 172.21.15.27:0/2633121364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d34143160 msgr2=0x7f2d34159030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:42:40.203 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.202+0000 7f2d3c55a700 1 --2- 172.21.15.27:0/2633121364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d34143160 0x7f2d34159030 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f2d2400c890 tx=0x7f2d240026a0 comp rx=0 tx=0).stop 2023-10-06T01:42:40.203 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.202+0000 7f2d3c55a700 1 -- 172.21.15.27:0/2633121364 shutdown_connections 2023-10-06T01:42:40.203 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.202+0000 7f2d3c55a700 1 --2- 172.21.15.27:0/2633121364 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f2d20066f30 0x7f2d200693e0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:40.203 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.202+0000 7f2d3c55a700 1 --2- 172.21.15.27:0/2633121364 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f2d34143160 0x7f2d34159030 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:40.204 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.202+0000 7f2d3c55a700 1 --2- 172.21.15.27:0/2633121364 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f2d340c4900 0x7f2d3415b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:40.204 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.202+0000 7f2d3c55a700 1 --2- 172.21.15.27:0/2633121364 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f2d34142d90 0x7f2d3413a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:42:40.204 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.202+0000 7f2d3c55a700 1 -- 172.21.15.27:0/2633121364 >> 172.21.15.27:0/2633121364 conn(0x7f2d340ce4d0 msgr2=0x7f2d340ce8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:42:40.204 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.203+0000 7f2d3c55a700 1 -- 172.21.15.27:0/2633121364 shutdown_connections 2023-10-06T01:42:40.204 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:42:40.203+0000 7f2d3c55a700 1 -- 172.21.15.27:0/2633121364 wait complete. 2023-10-06T01:42:40.204 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-06T01:42:40.214 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2023-10-06T01:42:40.058982+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2382936,"kb_used_data":4632,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752591784,"statfs":{"total":773094113280,"available":770653986816,"internally_reserved":0,"allocated":4743168,"data_stored":2679156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":-1,"num_object_clones":0,"num_object_copies":-2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":-4,"num_read_kb":-4,"num_write":-6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002145"},"pg_stats":[{"pgid":"1.5","version":"0'0","reported_seq":30,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-06T01:42:29.422519+0000","last_change":"2023-10-06T01:41:07.454295+0000","last_active":"2023-10-06T01:42:29.422519+0000","last_peered":"2023-10-06T01:42:29.422519+0000","last_clean":"2023-10-06T01:42:29.422519+0000","last_became_active":"2023-10-06T01:41:07.454034+0000","last_became_peered":"2023-10-06T01:41:07.454034+0000","last_unstale":"2023-10-06T01:42:29.422519+0000","last_undegraded":"2023-10-06T01:42:29.422519+0000","last_fullsized":"2023-10-06T01:42:29.422519+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T06:32:58.914183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":31,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-06T01:42:29.421800+0000","last_change":"2023-10-06T01:41:07.261955+0000","last_active":"2023-10-06T01:42:29.421800+0000","last_peered":"2023-10-06T01:42:29.421800+0000","last_clean":"2023-10-06T01:42:29.421800+0000","last_became_active":"2023-10-06T01:41:07.261692+0000","last_became_peered":"2023-10-06T01:41:07.261692+0000","last_unstale":"2023-10-06T01:42:29.421800+0000","last_undegraded":"2023-10-06T01:42:29.421800+0000","last_fullsized":"2023-10-06T01:42:29.421800+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":28,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-06T01:42:29.515137+0000","last_change":"2023-10-06T01:41:07.258362+0000","last_active":"2023-10-06T01:42:29.515137+0000","last_peered":"2023-10-06T01:42:29.515137+0000","last_clean":"2023-10-06T01:42:29.515137+0000","last_became_active":"2023-10-06T01:41:07.258055+0000","last_became_peered":"2023-10-06T01:41:07.258055+0000","last_unstale":"2023-10-06T01:42:29.515137+0000","last_undegraded":"2023-10-06T01:42:29.515137+0000","last_fullsized":"2023-10-06T01:42:29.515137+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"16'117","reported_seq":178,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-06T01:42:29.513225+0000","last_change":"2023-10-06T01:41:07.261164+0000","last_active":"2023-10-06T01:42:29.513225+0000","last_peered":"2023-10-06T01:42:29.513225+0000","last_clean":"2023-10-06T01:42:29.513225+0000","last_became_active":"2023-10-06T01:41:07.260896+0000","last_became_peered":"2023-10-06T01:41:07.260896+0000","last_unstale":"2023-10-06T01:42:29.513225+0000","last_undegraded":"2023-10-06T01:42:29.513225+0000","last_fullsized":"2023-10-06T01:42:29.513225+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:05:22.934401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":28,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-06T01:42:29.515247+0000","last_change":"2023-10-06T01:41:07.456511+0000","last_active":"2023-10-06T01:42:29.515247+0000","last_peered":"2023-10-06T01:42:29.515247+0000","last_clean":"2023-10-06T01:42:29.515247+0000","last_became_active":"2023-10-06T01:41:07.456024+0000","last_became_peered":"2023-10-06T01:41:07.456024+0000","last_unstale":"2023-10-06T01:42:29.515247+0000","last_undegraded":"2023-10-06T01:42:29.515247+0000","last_fullsized":"2023-10-06T01:42:29.515247+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T12:51:02.949231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":28,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-06T01:42:29.516306+0000","last_change":"2023-10-06T01:41:07.456273+0000","last_active":"2023-10-06T01:42:29.516306+0000","last_peered":"2023-10-06T01:42:29.516306+0000","last_clean":"2023-10-06T01:42:29.516306+0000","last_became_active":"2023-10-06T01:41:07.456000+0000","last_became_peered":"2023-10-06T01:41:07.456000+0000","last_unstale":"2023-10-06T01:42:29.516306+0000","last_undegraded":"2023-10-06T01:42:29.516306+0000","last_fullsized":"2023-10-06T01:42:29.516306+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:01:42.633388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":29,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-06T01:42:29.518007+0000","last_change":"2023-10-06T01:41:07.456688+0000","last_active":"2023-10-06T01:42:29.518007+0000","last_peered":"2023-10-06T01:42:29.518007+0000","last_clean":"2023-10-06T01:42:29.518007+0000","last_became_active":"2023-10-06T01:41:07.456304+0000","last_became_peered":"2023-10-06T01:41:07.456304+0000","last_unstale":"2023-10-06T01:42:29.518007+0000","last_undegraded":"2023-10-06T01:42:29.518007+0000","last_fullsized":"2023-10-06T01:42:29.518007+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":30,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-06T01:42:29.516302+0000","last_change":"2023-10-06T01:41:07.456388+0000","last_active":"2023-10-06T01:42:29.516302+0000","last_peered":"2023-10-06T01:42:29.516302+0000","last_clean":"2023-10-06T01:42:29.516302+0000","last_became_active":"2023-10-06T01:41:07.456045+0000","last_became_peered":"2023-10-06T01:41:07.456045+0000","last_unstale":"2023-10-06T01:42:29.516302+0000","last_undegraded":"2023-10-06T01:42:29.516302+0000","last_fullsized":"2023-10-06T01:42:29.516302+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:29:40.870791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":30,"reported_epoch":37,"state":"active+clean","last_fresh":"2023-10-06T01:42:29.517621+0000","last_change":"2023-10-06T01:41:07.258162+0000","last_active":"2023-10-06T01:42:29.517621+0000","last_peered":"2023-10-06T01:42:29.517621+0000","last_clean":"2023-10-06T01:42:29.517621+0000","last_became_active":"2023-10-06T01:41:07.257795+0000","last_became_peered":"2023-10-06T01:41:07.257795+0000","last_unstale":"2023-10-06T01:42:29.517621+0000","last_undegraded":"2023-10-06T01:42:29.517621+0000","last_fullsized":"2023-10-06T01:42:29.517621+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:41:06.244831+0000","last_clean_scrub_stamp":"2023-10-06T01:41:06.244831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T05:05:22.934401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574872,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298304,"kb_used_data":1016,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073536,"statfs":{"total":96636764160,"available":96331300864,"internally_reserved":0,"allocated":1040384,"data_stored":778347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 01:42:08 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.439}]},{"osd":1,"last update":"Fri Oct 6 01:42:08 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.52600000000000002}]},{"osd":2,"last update":"Fri Oct 6 01:42:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.54300000000000004}]},{"osd":3,"last update":"Fri Oct 6 01:42:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.626}]},{"osd":4,"last update":"Fri Oct 6 01:42:08 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.51200000000000001}]},{"osd":5,"last update":"Fri Oct 6 01:42:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.58499999999999996}]},{"osd":6,"last update":"Fri Oct 6 01:42:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.48699999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574872,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297720,"kb_used_data":432,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074120,"statfs":{"total":96636764160,"available":96331898880,"internally_reserved":0,"allocated":442368,"data_stored":186626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 01:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.57699999999999996}]},{"osd":1,"last update":"Fri Oct 6 01:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.53400000000000003}]},{"osd":2,"last update":"Fri Oct 6 01:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.51100000000000001}]},{"osd":3,"last update":"Fri Oct 6 01:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.65300000000000002}]},{"osd":4,"last update":"Fri Oct 6 01:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.61699999999999999}]},{"osd":5,"last update":"Fri Oct 6 01:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.625},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.66700000000000004}]},{"osd":7,"last update":"Fri Oct 6 01:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.59399999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574870,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298304,"kb_used_data":1016,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073536,"statfs":{"total":96636764160,"available":96331300864,"internally_reserved":0,"allocated":1040384,"data_stored":778347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.30099999999999999}]},{"osd":2,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.31900000000000001}]},{"osd":3,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.34799999999999998}]},{"osd":4,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.498}]},{"osd":5,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.65500000000000003}]},{"osd":6,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.56799999999999995}]},{"osd":7,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.63}]}]},{"osd":0,"up_from":13,"seq":55834574870,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297720,"kb_used_data":432,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074120,"statfs":{"total":96636764160,"available":96331898880,"internally_reserved":0,"allocated":442368,"data_stored":186626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.495,"5min":0.495,"15min":0.495},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"last":0.23200000000000001}]},{"osd":2,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"last":0.28799999999999998}]},{"osd":3,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.439,"5min":0.439,"15min":0.439},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.36599999999999999}]},{"osd":4,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.439}]},{"osd":5,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.45300000000000001}]},{"osd":6,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.41799999999999998}]},{"osd":7,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.53400000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297724,"kb_used_data":436,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074116,"statfs":{"total":96636764160,"available":96331894784,"internally_reserved":0,"allocated":446464,"data_stored":187979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.497,"5min":0.497,"15min":0.497},"last":0.36599999999999999}]},{"osd":1,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.317},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.29799999999999999}]},{"osd":3,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.38100000000000001}]},{"osd":4,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.51100000000000001}]},{"osd":5,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.46999999999999997}]},{"osd":6,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.439}]},{"osd":7,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.52300000000000002}]}]},{"osd":3,"up_from":13,"seq":55834574870,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297720,"kb_used_data":432,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074120,"statfs":{"total":96636764160,"available":96331898880,"internally_reserved":0,"allocated":442368,"data_stored":186626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.31,"15min":0.31},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"last":0.38400000000000001}]},{"osd":1,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.40400000000000003}]},{"osd":2,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"last":0.47899999999999998}]},{"osd":4,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.61099999999999999}]},{"osd":5,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.67900000000000005}]},{"osd":6,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.628},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.50900000000000001}]},{"osd":7,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.69299999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574871,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297724,"kb_used_data":436,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074116,"statfs":{"total":96636764160,"available":96331894784,"internally_reserved":0,"allocated":446464,"data_stored":187979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.42099999999999999}]},{"osd":1,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.498},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.45500000000000002}]},{"osd":2,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.55300000000000005}]},{"osd":3,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.57299999999999995}]},{"osd":5,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.65600000000000003}]},{"osd":6,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.64500000000000002}]},{"osd":7,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.66600000000000004}]}]},{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297720,"kb_used_data":432,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074120,"statfs":{"total":96636764160,"available":96331898880,"internally_reserved":0,"allocated":442368,"data_stored":186626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.439},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.335,"5min":1.335,"15min":1.335},"last":0.54700000000000004}]},{"osd":1,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":2.4849999999999999,"5min":2.4849999999999999,"15min":2.4849999999999999},"last":0.53200000000000003}]},{"osd":2,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.8280000000000001,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":0.45500000000000002}]},{"osd":3,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.9830000000000001,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.60099999999999998}]},{"osd":4,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.47999999999999998}]},{"osd":6,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.501},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.51400000000000001}]},{"osd":7,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.62}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-06T01:42:40.219 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2023-10-06T01:41:06.244831+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=1, tm_min=41, tm_sec=6, tm_wday=4, tm_yday=279, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=1, tm_min=42, tm_sec=34, tm_wday=4, tm_yday=279, tm_isdst=0) 2023-10-06T01:42:40.219 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2023-10-06T01:41:06.244831+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=1, tm_min=41, tm_sec=6, tm_wday=4, tm_yday=279, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=1, tm_min=42, tm_sec=34, tm_wday=4, tm_yday=279, tm_isdst=0) 2023-10-06T01:42:40.219 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2023-10-06T01:41:06.244831+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=1, tm_min=41, tm_sec=6, tm_wday=4, tm_yday=279, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=1, tm_min=42, tm_sec=34, tm_wday=4, tm_yday=279, tm_isdst=0) 2023-10-06T01:42:40.219 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-06T01:41:06.244831+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=1, tm_min=41, tm_sec=6, tm_wday=4, tm_yday=279, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=1, tm_min=42, tm_sec=34, tm_wday=4, tm_yday=279, tm_isdst=0) 2023-10-06T01:42:40.219 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2023-10-06T01:41:06.244831+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=1, tm_min=41, tm_sec=6, tm_wday=4, tm_yday=279, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=1, tm_min=42, tm_sec=34, tm_wday=4, tm_yday=279, tm_isdst=0) 2023-10-06T01:42:40.220 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-06T01:41:06.244831+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=1, tm_min=41, tm_sec=6, tm_wday=4, tm_yday=279, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=1, tm_min=42, tm_sec=34, tm_wday=4, tm_yday=279, tm_isdst=0) 2023-10-06T01:42:40.220 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2023-10-06T01:41:06.244831+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=1, tm_min=41, tm_sec=6, tm_wday=4, tm_yday=279, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=1, tm_min=42, tm_sec=34, tm_wday=4, tm_yday=279, tm_isdst=0) 2023-10-06T01:42:40.220 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2023-10-06T01:41:06.244831+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=1, tm_min=41, tm_sec=6, tm_wday=4, tm_yday=279, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=1, tm_min=42, tm_sec=34, tm_wday=4, tm_yday=279, tm_isdst=0) 2023-10-06T01:42:40.220 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2023-10-06T01:41:06.244831+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=1, tm_min=41, tm_sec=6, tm_wday=4, tm_yday=279, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=6, tm_hour=1, tm_min=42, tm_sec=34, tm_wday=4, tm_yday=279, tm_isdst=0) 2023-10-06T01:42:40.220 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-06T01:42:48.719 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T01:42:48.722 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T01:43:00.221 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-06T01:43:00.355 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.354+0000 7fa7fe237700 1 Processor -- start 2023-10-06T01:43:00.355 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.354+0000 7fa7fe237700 1 -- start start 2023-10-06T01:43:00.357 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.354+0000 7fa7fe237700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa7f8142d90 0x7fa7f806a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:43:00.357 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.354+0000 7fa7fe237700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fa7f80cc130 0x7fa7f80cc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:43:00.357 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.354+0000 7fa7fe237700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7f8069f50 con 0x7fa7f8143160 2023-10-06T01:43:00.357 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.354+0000 7fa7fe237700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7f80ccaf0 con 0x7fa7f8142d90 2023-10-06T01:43:00.358 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.354+0000 7fa7fe237700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7f806a920 con 0x7fa7f80cc130 2023-10-06T01:43:00.358 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.355+0000 7fa7f77fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa7f8142d90 0x7fa7f806a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:43:00.358 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.355+0000 7fa7f77fe700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fa7f8142d90 0x7fa7f806a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38666/0 (socket says 172.21.15.27:38666) 2023-10-06T01:43:00.358 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.355+0000 7fa7f77fe700 1 -- 172.21.15.27:0/2389182687 learned_addr learned my addr 172.21.15.27:0/2389182687 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:43:00.358 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.355+0000 7fa7f7fff700 1 --2- 172.21.15.27:0/2389182687 >> v2:172.21.15.124:3300/0 conn(0x7fa7f80cc130 0x7fa7f80cc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:43:00.358 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.355+0000 7fa7f77fe700 1 -- 172.21.15.27:0/2389182687 >> v2:172.21.15.124:3300/0 conn(0x7fa7f80cc130 msgr2=0x7fa7f80cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:43:00.358 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.355+0000 7fa7f77fe700 1 --2- 172.21.15.27:0/2389182687 >> v2:172.21.15.124:3300/0 conn(0x7fa7f80cc130 0x7fa7f80cc5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:43:00.358 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.355+0000 7fa7f77fe700 1 -- 172.21.15.27:0/2389182687 >> v1:172.21.15.124:6789/0 conn(0x7fa7f8143160 legacy=0x7fa7f80c43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:43:00.358 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.355+0000 7fa7f77fe700 1 -- 172.21.15.27:0/2389182687 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f8069340 con 0x7fa7f8142d90 2023-10-06T01:43:00.358 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.355+0000 7fa7f77fe700 1 --2- 172.21.15.27:0/2389182687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7f8142d90 0x7fa7f806a3e0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fa7f80b1010 tx=0x7fa7e8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a96f506ef07d323 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:43:00.358 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.355+0000 7fa7f67fc700 1 -- 172.21.15.27:0/2389182687 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e800b860 con 0x7fa7f8142d90 2023-10-06T01:43:00.358 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.355+0000 7fa7f67fc700 1 -- 172.21.15.27:0/2389182687 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7e800b9c0 con 0x7fa7f8142d90 2023-10-06T01:43:00.359 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.355+0000 7fa7f67fc700 1 -- 172.21.15.27:0/2389182687 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e800bd60 con 0x7fa7f8142d90 2023-10-06T01:43:00.359 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.356+0000 7fa7fe237700 1 -- 172.21.15.27:0/2389182687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7f8142d90 msgr2=0x7fa7f806a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:43:00.359 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.356+0000 7fa7fe237700 1 --2- 172.21.15.27:0/2389182687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7f8142d90 0x7fa7f806a3e0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fa7f80b1010 tx=0x7fa7e8009250 comp rx=0 tx=0).stop 2023-10-06T01:43:00.359 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.356+0000 7fa7fe237700 1 -- 172.21.15.27:0/2389182687 shutdown_connections 2023-10-06T01:43:00.359 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.356+0000 7fa7fe237700 1 --2- 172.21.15.27:0/2389182687 >> v2:172.21.15.124:3300/0 conn(0x7fa7f80cc130 0x7fa7f80cc5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:43:00.359 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.356+0000 7fa7fe237700 1 --2- 172.21.15.27:0/2389182687 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7f8142d90 0x7fa7f806a3e0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:43:00.359 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.356+0000 7fa7fe237700 1 -- 172.21.15.27:0/2389182687 >> 172.21.15.27:0/2389182687 conn(0x7fa7f80ce4d0 msgr2=0x7fa7f80ce8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:43:00.359 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.356+0000 7fa7fe237700 1 -- 172.21.15.27:0/2389182687 shutdown_connections 2023-10-06T01:43:00.359 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.356+0000 7fa7fe237700 1 -- 172.21.15.27:0/2389182687 wait complete. 2023-10-06T01:43:00.361 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.357+0000 7fa7fe237700 1 Processor -- start 2023-10-06T01:43:00.361 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.357+0000 7fa7fe237700 1 -- start start 2023-10-06T01:43:00.361 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.358+0000 7fa7fe237700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7f80cc130 0x7fa7f813b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:43:00.361 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.358+0000 7fa7fe237700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa7f8142d90 0x7fa7f815b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:43:00.361 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.358+0000 7fa7fe237700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa7f8143160 0x7fa7f8141c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:43:00.361 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.358+0000 7fa7fe237700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f81425f0 con 0x7fa7f80cc130 2023-10-06T01:43:00.361 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.358+0000 7fa7fe237700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f815f320 con 0x7fa7f8143160 2023-10-06T01:43:00.362 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.358+0000 7fa7fe237700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f815e9f0 con 0x7fa7f8142d90 2023-10-06T01:43:00.362 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.358+0000 7fa7f77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7f80cc130 0x7fa7f813b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:43:00.362 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.358+0000 7fa7f77fe700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7f80cc130 0x7fa7f813b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38674/0 (socket says 172.21.15.27:38674) 2023-10-06T01:43:00.362 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.358+0000 7fa7f77fe700 1 -- 172.21.15.27:0/939259965 learned_addr learned my addr 172.21.15.27:0/939259965 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:43:00.362 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.359+0000 7fa7f77fe700 1 -- 172.21.15.27:0/939259965 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa7f8142d90 msgr2=0x7fa7f815b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:43:00.362 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.359+0000 7fa7f7fff700 1 --2- 172.21.15.27:0/939259965 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa7f8143160 0x7fa7f8141c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:43:00.362 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.359+0000 7fa7f77fe700 1 --2- 172.21.15.27:0/939259965 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa7f8142d90 0x7fa7f815b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:43:00.362 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.359+0000 7fa7f77fe700 1 -- 172.21.15.27:0/939259965 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa7f8143160 msgr2=0x7fa7f8141c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:43:00.362 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.359+0000 7fa7f77fe700 1 --2- 172.21.15.27:0/939259965 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa7f8143160 0x7fa7f8141c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:43:00.362 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.359+0000 7fa7f77fe700 1 -- 172.21.15.27:0/939259965 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e80094f0 con 0x7fa7f80cc130 2023-10-06T01:43:00.362 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.359+0000 7fa7f7fff700 1 --2- 172.21.15.27:0/939259965 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa7f8143160 0x7fa7f8141c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:43:00.362 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.359+0000 7fa7f77fe700 1 --2- 172.21.15.27:0/939259965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7f80cc130 0x7fa7f813b160 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fa7f8068380 tx=0x7fa7e8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:43:00.363 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.359+0000 7fa7df7fe700 1 -- 172.21.15.27:0/939259965 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e8011e60 con 0x7fa7f80cc130 2023-10-06T01:43:00.363 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.359+0000 7fa7df7fe700 1 -- 172.21.15.27:0/939259965 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7e801a630 con 0x7fa7f80cc130 2023-10-06T01:43:00.363 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.359+0000 7fa7fe237700 1 -- 172.21.15.27:0/939259965 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7f815ecd0 con 0x7fa7f80cc130 2023-10-06T01:43:00.363 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.359+0000 7fa7df7fe700 1 -- 172.21.15.27:0/939259965 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e801a9f0 con 0x7fa7f80cc130 2023-10-06T01:43:00.363 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.359+0000 7fa7fe237700 1 -- 172.21.15.27:0/939259965 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7f813b6a0 con 0x7fa7f80cc130 2023-10-06T01:43:00.363 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.360+0000 7fa7fe237700 1 -- 172.21.15.27:0/939259965 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7f8064050 con 0x7fa7f80cc130 2023-10-06T01:43:00.364 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.360+0000 7fa7df7fe700 1 -- 172.21.15.27:0/939259965 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fa7e802e430 con 0x7fa7f80cc130 2023-10-06T01:43:00.364 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.361+0000 7fa7df7fe700 1 --2- 172.21.15.27:0/939259965 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa7e4066f30 0x7fa7e40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:43:00.364 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.361+0000 7fa7df7fe700 1 -- 172.21.15.27:0/939259965 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5292+0+0 (secure 0 0 0) 0x7fa7e8085fe0 con 0x7fa7f80cc130 2023-10-06T01:43:00.364 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.361+0000 7fa7f6ffd700 1 --2- 172.21.15.27:0/939259965 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa7e4066f30 0x7fa7e40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:43:00.364 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.361+0000 7fa7f6ffd700 1 --2- 172.21.15.27:0/939259965 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa7e4066f30 0x7fa7e40693e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa7f815d220 tx=0x7fa7ec007b40 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:43:00.366 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.365+0000 7fa7df7fe700 1 -- 172.21.15.27:0/939259965 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa7e80533f0 con 0x7fa7f80cc130 2023-10-06T01:43:00.557 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.556+0000 7fa7fe237700 1 -- 172.21.15.27:0/939259965 --> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa7f8069620 con 0x7fa7e4066f30 2023-10-06T01:43:00.561 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.560+0000 7fa7df7fe700 1 -- 172.21.15.27:0/939259965 <== mgr.4100 v2:172.21.15.124:6832/90278 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+73314 (secure 0 0 0) 0x7fa7f8069620 con 0x7fa7e4066f30 2023-10-06T01:43:00.562 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:43:00.564 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.562+0000 7fa7fe237700 1 -- 172.21.15.27:0/939259965 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa7e4066f30 msgr2=0x7fa7e40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:43:00.564 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.562+0000 7fa7fe237700 1 --2- 172.21.15.27:0/939259965 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa7e4066f30 0x7fa7e40693e0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fa7f815d220 tx=0x7fa7ec007b40 comp rx=0 tx=0).stop 2023-10-06T01:43:00.565 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.562+0000 7fa7fe237700 1 -- 172.21.15.27:0/939259965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7f80cc130 msgr2=0x7fa7f813b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:43:00.565 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.562+0000 7fa7fe237700 1 --2- 172.21.15.27:0/939259965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7f80cc130 0x7fa7f813b160 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fa7f8068380 tx=0x7fa7e8017000 comp rx=0 tx=0).stop 2023-10-06T01:43:00.565 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.562+0000 7fa7fe237700 1 -- 172.21.15.27:0/939259965 shutdown_connections 2023-10-06T01:43:00.565 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.562+0000 7fa7fe237700 1 --2- 172.21.15.27:0/939259965 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fa7e4066f30 0x7fa7e40693e0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:43:00.565 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.562+0000 7fa7fe237700 1 --2- 172.21.15.27:0/939259965 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fa7f8143160 0x7fa7f8141c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:43:00.565 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.562+0000 7fa7fe237700 1 --2- 172.21.15.27:0/939259965 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fa7f80cc130 0x7fa7f813b160 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:43:00.565 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.562+0000 7fa7fe237700 1 --2- 172.21.15.27:0/939259965 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fa7f8142d90 0x7fa7f815b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:43:00.566 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.562+0000 7fa7fe237700 1 -- 172.21.15.27:0/939259965 >> 172.21.15.27:0/939259965 conn(0x7fa7f80ce4d0 msgr2=0x7fa7f80b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:43:00.566 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.562+0000 7fa7fe237700 1 -- 172.21.15.27:0/939259965 shutdown_connections 2023-10-06T01:43:00.566 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.562+0000 7fa7fe237700 1 -- 172.21.15.27:0/939259965 wait complete. 2023-10-06T01:43:00.566 INFO:teuthology.orchestra.run.smithi027.stderr:dumped all 2023-10-06T01:43:00.575 INFO:teuthology.orchestra.run.smithi027.stdout:{"pg_ready":true,"pg_map":{"version":88,"stamp":"2023-10-06T01:43:00.062996+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":2382936,"kb_used_data":4632,"kb_used_omap":0,"kb_used_meta":2378240,"kb_avail":752591784,"statfs":{"total":773094113280,"available":770653986816,"internally_reserved":0,"allocated":4743168,"data_stored":2679156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":2435317760},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002453"},"pg_stats":[{"pgid":"1.5","version":"0'0","reported_seq":41,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-06T01:42:40.667863+0000","last_change":"2023-10-06T01:42:40.667835+0000","last_active":"2023-10-06T01:42:40.667863+0000","last_peered":"2023-10-06T01:42:40.667863+0000","last_clean":"2023-10-06T01:42:40.667863+0000","last_became_active":"2023-10-06T01:41:07.454034+0000","last_became_peered":"2023-10-06T01:41:07.454034+0000","last_unstale":"2023-10-06T01:42:40.667863+0000","last_undegraded":"2023-10-06T01:42:40.667863+0000","last_fullsized":"2023-10-06T01:42:40.667863+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:40.667730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:40.667730+0000","last_clean_scrub_stamp":"2023-10-06T01:42:40.667730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:37:08.168353+0000","scrub_duration":0.0018614599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":42,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-06T01:42:40.928992+0000","last_change":"2023-10-06T01:42:40.928968+0000","last_active":"2023-10-06T01:42:40.928992+0000","last_peered":"2023-10-06T01:42:40.928992+0000","last_clean":"2023-10-06T01:42:40.928992+0000","last_became_active":"2023-10-06T01:41:07.261692+0000","last_became_peered":"2023-10-06T01:41:07.261692+0000","last_unstale":"2023-10-06T01:42:40.928992+0000","last_undegraded":"2023-10-06T01:42:40.928992+0000","last_fullsized":"2023-10-06T01:42:40.928992+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:40.928865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:40.928865+0000","last_clean_scrub_stamp":"2023-10-06T01:42:40.928865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T09:37:08.429488+0000","scrub_duration":0.001915527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-06T01:42:40.570158+0000","last_change":"2023-10-06T01:42:40.570134+0000","last_active":"2023-10-06T01:42:40.570158+0000","last_peered":"2023-10-06T01:42:40.570158+0000","last_clean":"2023-10-06T01:42:40.570158+0000","last_became_active":"2023-10-06T01:41:07.258055+0000","last_became_peered":"2023-10-06T01:41:07.258055+0000","last_unstale":"2023-10-06T01:42:40.570158+0000","last_undegraded":"2023-10-06T01:42:40.570158+0000","last_fullsized":"2023-10-06T01:42:40.570158+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:40.570044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:40.570044+0000","last_clean_scrub_stamp":"2023-10-06T01:42:40.570044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:05:48.903461+0000","scrub_duration":0.0018137520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"38'119","reported_seq":191,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-06T01:42:48.357335+0000","last_change":"2023-10-06T01:42:48.357309+0000","last_active":"2023-10-06T01:42:48.357335+0000","last_peered":"2023-10-06T01:42:48.357335+0000","last_clean":"2023-10-06T01:42:48.357335+0000","last_became_active":"2023-10-06T01:41:07.260896+0000","last_became_peered":"2023-10-06T01:41:07.260896+0000","last_unstale":"2023-10-06T01:42:48.357335+0000","last_undegraded":"2023-10-06T01:42:48.357335+0000","last_fullsized":"2023-10-06T01:42:48.357335+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"38'119","last_scrub_stamp":"2023-10-06T01:42:48.357188+0000","last_deep_scrub":"38'119","last_deep_scrub_stamp":"2023-10-06T01:42:48.357188+0000","last_clean_scrub_stamp":"2023-10-06T01:42:48.357188+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T07:11:51.782274+0000","scrub_duration":0.0075010659999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-06T01:42:40.297837+0000","last_change":"2023-10-06T01:42:40.297812+0000","last_active":"2023-10-06T01:42:40.297837+0000","last_peered":"2023-10-06T01:42:40.297837+0000","last_clean":"2023-10-06T01:42:40.297837+0000","last_became_active":"2023-10-06T01:41:07.456024+0000","last_became_peered":"2023-10-06T01:41:07.456024+0000","last_unstale":"2023-10-06T01:42:40.297837+0000","last_undegraded":"2023-10-06T01:42:40.297837+0000","last_fullsized":"2023-10-06T01:42:40.297837+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:40.297707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:40.297707+0000","last_clean_scrub_stamp":"2023-10-06T01:42:40.297707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:37:37.191450+0000","scrub_duration":0.0016917900000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":39,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-06T01:42:41.271605+0000","last_change":"2023-10-06T01:42:41.271580+0000","last_active":"2023-10-06T01:42:41.271605+0000","last_peered":"2023-10-06T01:42:41.271605+0000","last_clean":"2023-10-06T01:42:41.271605+0000","last_became_active":"2023-10-06T01:41:07.456000+0000","last_became_peered":"2023-10-06T01:41:07.456000+0000","last_unstale":"2023-10-06T01:42:41.271605+0000","last_undegraded":"2023-10-06T01:42:41.271605+0000","last_fullsized":"2023-10-06T01:42:41.271605+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:41.271490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:41.271490+0000","last_clean_scrub_stamp":"2023-10-06T01:42:41.271490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T02:15:01.038901+0000","scrub_duration":0.0020087239999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":40,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-06T01:42:40.122711+0000","last_change":"2023-10-06T01:42:40.122675+0000","last_active":"2023-10-06T01:42:40.122711+0000","last_peered":"2023-10-06T01:42:40.122711+0000","last_clean":"2023-10-06T01:42:40.122711+0000","last_became_active":"2023-10-06T01:41:07.456304+0000","last_became_peered":"2023-10-06T01:41:07.456304+0000","last_unstale":"2023-10-06T01:42:40.122711+0000","last_undegraded":"2023-10-06T01:42:40.122711+0000","last_fullsized":"2023-10-06T01:42:40.122711+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:40.122573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:40.122573+0000","last_clean_scrub_stamp":"2023-10-06T01:42:40.122573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:37:37.016316+0000","scrub_duration":0.0015491529999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":41,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-06T01:42:51.952932+0000","last_change":"2023-10-06T01:42:51.952891+0000","last_active":"2023-10-06T01:42:51.952932+0000","last_peered":"2023-10-06T01:42:51.952932+0000","last_clean":"2023-10-06T01:42:51.952932+0000","last_became_active":"2023-10-06T01:41:07.456045+0000","last_became_peered":"2023-10-06T01:41:07.456045+0000","last_unstale":"2023-10-06T01:42:51.952932+0000","last_undegraded":"2023-10-06T01:42:51.952932+0000","last_fullsized":"2023-10-06T01:42:51.952932+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:51.952764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:51.952764+0000","last_clean_scrub_stamp":"2023-10-06T01:42:51.952764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T04:37:48.846507+0000","scrub_duration":0.0022124520000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":41,"reported_epoch":38,"state":"active+clean","last_fresh":"2023-10-06T01:42:44.967842+0000","last_change":"2023-10-06T01:42:44.967813+0000","last_active":"2023-10-06T01:42:44.967842+0000","last_peered":"2023-10-06T01:42:44.967842+0000","last_clean":"2023-10-06T01:42:44.967842+0000","last_became_active":"2023-10-06T01:41:07.257795+0000","last_became_peered":"2023-10-06T01:41:07.257795+0000","last_unstale":"2023-10-06T01:42:44.967842+0000","last_undegraded":"2023-10-06T01:42:44.967842+0000","last_fullsized":"2023-10-06T01:42:44.967842+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-06T01:42:44.967708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-06T01:42:44.967708+0000","last_clean_scrub_stamp":"2023-10-06T01:42:44.967708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-07T08:05:53.301126+0000","scrub_duration":0.0021211619999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574876,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298304,"kb_used_data":1016,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073536,"statfs":{"total":96636764160,"available":96331300864,"internally_reserved":0,"allocated":1040384,"data_stored":778347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 01:42:08 2023","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.53600000000000003}]},{"osd":1,"last update":"Fri Oct 6 01:42:08 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.57899999999999996}]},{"osd":2,"last update":"Fri Oct 6 01:42:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.56899999999999995}]},{"osd":3,"last update":"Fri Oct 6 01:42:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.51700000000000002}]},{"osd":4,"last update":"Fri Oct 6 01:42:08 2023","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.44600000000000001}]},{"osd":5,"last update":"Fri Oct 6 01:42:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.60799999999999998}]},{"osd":6,"last update":"Fri Oct 6 01:42:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.58599999999999997}]}]},{"osd":6,"up_from":13,"seq":55834574876,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297720,"kb_used_data":432,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074120,"statfs":{"total":96636764160,"available":96331898880,"internally_reserved":0,"allocated":442368,"data_stored":186626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 01:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.57399999999999995}]},{"osd":1,"last update":"Fri Oct 6 01:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.59999999999999998}]},{"osd":2,"last update":"Fri Oct 6 01:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.58799999999999997}]},{"osd":3,"last update":"Fri Oct 6 01:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.52800000000000002}]},{"osd":4,"last update":"Fri Oct 6 01:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.36099999999999999}]},{"osd":5,"last update":"Fri Oct 6 01:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.53900000000000003}]},{"osd":7,"last update":"Fri Oct 6 01:42:06 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.55700000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574874,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298304,"kb_used_data":1016,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94073536,"statfs":{"total":96636764160,"available":96331300864,"internally_reserved":0,"allocated":1040384,"data_stored":778347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.55900000000000005}]},{"osd":2,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.57599999999999996}]},{"osd":3,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.496}]},{"osd":4,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.754}]},{"osd":5,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.69799999999999995}]},{"osd":6,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.67500000000000004}]},{"osd":7,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"last":0.65000000000000002}]}]},{"osd":0,"up_from":13,"seq":55834574874,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297720,"kb_used_data":432,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074120,"statfs":{"total":96636764160,"available":96331898880,"internally_reserved":0,"allocated":442368,"data_stored":186626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.495,"5min":0.495,"15min":0.495},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"last":0.39000000000000001}]},{"osd":2,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"last":0.497}]},{"osd":3,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.439,"5min":0.439,"15min":0.439},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.48499999999999999}]},{"osd":4,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.58499999999999996}]},{"osd":5,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.67500000000000004}]},{"osd":6,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.70999999999999996}]},{"osd":7,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.622}]}]},{"osd":2,"up_from":13,"seq":55834574875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297724,"kb_used_data":436,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074116,"statfs":{"total":96636764160,"available":96331894784,"internally_reserved":0,"allocated":446464,"data_stored":187979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.497,"5min":0.497,"15min":0.497},"last":0.309}]},{"osd":1,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.42299999999999999}]},{"osd":3,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.47399999999999998}]},{"osd":4,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.57599999999999996}]},{"osd":5,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.64400000000000002}]},{"osd":6,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.63400000000000001}]},{"osd":7,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.61799999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574874,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297720,"kb_used_data":432,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074120,"statfs":{"total":96636764160,"available":96331898880,"internally_reserved":0,"allocated":442368,"data_stored":186626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.31,"5min":0.31,"15min":0.31},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"last":0.52900000000000003}]},{"osd":1,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"last":0.61599999999999999}]},{"osd":2,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"last":0.437},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"last":0.67100000000000004}]},{"osd":4,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.58499999999999996}]},{"osd":5,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.73199999999999998}]},{"osd":6,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.69699999999999995}]},{"osd":7,"last update":"Fri Oct 6 01:42:07 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.63500000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297724,"kb_used_data":436,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074116,"statfs":{"total":96636764160,"available":96331894784,"internally_reserved":0,"allocated":446464,"data_stored":187979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.48699999999999999}]},{"osd":1,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.51600000000000001}]},{"osd":2,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.52900000000000003}]},{"osd":3,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.505}]},{"osd":5,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.376}]},{"osd":6,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.436},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.35999999999999999}]},{"osd":7,"last update":"Fri Oct 6 01:42:09 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.38700000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574873,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297720,"kb_used_data":432,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":94074120,"statfs":{"total":96636764160,"available":96331898880,"internally_reserved":0,"allocated":442368,"data_stored":186626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.3979999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.335,"5min":1.335,"15min":1.335},"last":0.59599999999999997}]},{"osd":1,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":2.4849999999999999,"5min":2.4849999999999999,"15min":2.4849999999999999},"last":0.53000000000000003}]},{"osd":2,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":1.8280000000000001,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":0.57799999999999996}]},{"osd":3,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":1.9830000000000001,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.62}]},{"osd":4,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.60499999999999998}]},{"osd":6,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.441}]},{"osd":7,"last update":"Fri Oct 6 01:42:05 2023","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.498}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-06T01:43:00.578 DEBUG:teuthology.orchestra.run.smithi027:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-06T01:43:00.696 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.694+0000 7fbe8bfac700 1 Processor -- start 2023-10-06T01:43:00.696 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.694+0000 7fbe8bfac700 1 -- start start 2023-10-06T01:43:00.698 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.695+0000 7fbe8bfac700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbe84159500 0x7fbe841598d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:43:00.698 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.695+0000 7fbe8bfac700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7fbe840b3fd0 0x7fbe8415d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:43:00.698 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.695+0000 7fbe8bfac700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbe8415d8e0 0x7fbe840cf6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:43:00.699 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.695+0000 7fbe8bfac700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe8415dd20 con 0x7fbe8415d8e0 2023-10-06T01:43:00.699 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.695+0000 7fbe8bfac700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe840b1110 con 0x7fbe840b3fd0 2023-10-06T01:43:00.699 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.695+0000 7fbe8bfac700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe840b13b0 con 0x7fbe84159500 2023-10-06T01:43:00.699 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.695+0000 7fbe8a549700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbe8415d8e0 0x7fbe840cf6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:43:00.699 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.696+0000 7fbe8a549700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7fbe8415d8e0 0x7fbe840cf6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38684/0 (socket says 172.21.15.27:38684) 2023-10-06T01:43:00.699 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.696+0000 7fbe8a549700 1 -- 172.21.15.27:0/454192082 learned_addr learned my addr 172.21.15.27:0/454192082 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:43:00.699 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.696+0000 7fbe89d48700 1 --2- 172.21.15.27:0/454192082 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbe84159500 0x7fbe841598d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:43:00.699 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.696+0000 7fbe89547700 1 --2- 172.21.15.27:0/454192082 >> v2:172.21.15.124:3300/0 conn(0x7fbe840b3fd0 0x7fbe8415d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:43:00.699 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.696+0000 7fbe8a549700 1 -- 172.21.15.27:0/454192082 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbe84159500 msgr2=0x7fbe841598d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:43:00.699 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.696+0000 7fbe8a549700 1 --2- 172.21.15.27:0/454192082 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbe84159500 0x7fbe841598d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:43:00.699 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.696+0000 7fbe8a549700 1 -- 172.21.15.27:0/454192082 >> v2:172.21.15.124:3300/0 conn(0x7fbe840b3fd0 msgr2=0x7fbe8415d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:43:00.699 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.696+0000 7fbe8a549700 1 --2- 172.21.15.27:0/454192082 >> v2:172.21.15.124:3300/0 conn(0x7fbe840b3fd0 0x7fbe8415d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:43:00.699 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.696+0000 7fbe8a549700 1 -- 172.21.15.27:0/454192082 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe840b1690 con 0x7fbe8415d8e0 2023-10-06T01:43:00.700 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.696+0000 7fbe89547700 1 --2- 172.21.15.27:0/454192082 >> v2:172.21.15.124:3300/0 conn(0x7fbe840b3fd0 0x7fbe8415d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-06T01:43:00.700 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.696+0000 7fbe8a549700 1 --2- 172.21.15.27:0/454192082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe8415d8e0 0x7fbe840cf6c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fbe840891f0 tx=0x7fbe7c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9529e2438d1e3df server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:43:00.700 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.696+0000 7fbe88d46700 1 -- 172.21.15.27:0/454192082 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe7c00e070 con 0x7fbe8415d8e0 2023-10-06T01:43:00.700 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.696+0000 7fbe88d46700 1 -- 172.21.15.27:0/454192082 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe7c00ba50 con 0x7fbe8415d8e0 2023-10-06T01:43:00.700 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.696+0000 7fbe88d46700 1 -- 172.21.15.27:0/454192082 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe7c0092e0 con 0x7fbe8415d8e0 2023-10-06T01:43:00.701 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.697+0000 7fbe8bfac700 1 -- 172.21.15.27:0/454192082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe8415d8e0 msgr2=0x7fbe840cf6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:43:00.701 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.697+0000 7fbe8bfac700 1 --2- 172.21.15.27:0/454192082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe8415d8e0 0x7fbe840cf6c0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fbe840891f0 tx=0x7fbe7c009840 comp rx=0 tx=0).stop 2023-10-06T01:43:00.701 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.697+0000 7fbe8bfac700 1 -- 172.21.15.27:0/454192082 shutdown_connections 2023-10-06T01:43:00.701 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.697+0000 7fbe8bfac700 1 --2- 172.21.15.27:0/454192082 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe8415d8e0 0x7fbe840cf6c0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:43:00.701 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.697+0000 7fbe8bfac700 1 --2- 172.21.15.27:0/454192082 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbe84159500 0x7fbe841598d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:43:00.701 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.697+0000 7fbe8bfac700 1 --2- 172.21.15.27:0/454192082 >> v2:172.21.15.124:3300/0 conn(0x7fbe840b3fd0 0x7fbe8415d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:43:00.701 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.697+0000 7fbe8bfac700 1 -- 172.21.15.27:0/454192082 >> 172.21.15.27:0/454192082 conn(0x7fbe84067630 msgr2=0x7fbe84069a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:43:00.701 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.697+0000 7fbe8bfac700 1 -- 172.21.15.27:0/454192082 shutdown_connections 2023-10-06T01:43:00.701 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.697+0000 7fbe8bfac700 1 -- 172.21.15.27:0/454192082 wait complete. 2023-10-06T01:43:00.704 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.698+0000 7fbe8bfac700 1 Processor -- start 2023-10-06T01:43:00.704 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.698+0000 7fbe8bfac700 1 -- start start 2023-10-06T01:43:00.704 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.699+0000 7fbe8bfac700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbe840b3fd0 0x7fbe8413e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:43:00.704 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.699+0000 7fbe8bfac700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbe84159500 0x7fbe8416d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:43:00.704 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.699+0000 7fbe8bfac700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe8415d8e0 0x7fbe8416db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:43:00.704 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.699+0000 7fbe8bfac700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe840a70a0 con 0x7fbe8415d8e0 2023-10-06T01:43:00.704 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.699+0000 7fbe8bfac700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe840a7380 con 0x7fbe84159500 2023-10-06T01:43:00.704 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.699+0000 7fbe8bfac700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe840a7660 con 0x7fbe840b3fd0 2023-10-06T01:43:00.704 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.699+0000 7fbe8a549700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe8415d8e0 0x7fbe8416db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:43:00.705 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.699+0000 7fbe8a549700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe8415d8e0 0x7fbe8416db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38688/0 (socket says 172.21.15.27:38688) 2023-10-06T01:43:00.705 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.699+0000 7fbe8a549700 1 -- 172.21.15.27:0/3511508919 learned_addr learned my addr 172.21.15.27:0/3511508919 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:43:00.705 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.699+0000 7fbe89547700 1 --2- 172.21.15.27:0/3511508919 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbe84159500 0x7fbe8416d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:43:00.705 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.699+0000 7fbe89d48700 1 --2- 172.21.15.27:0/3511508919 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbe840b3fd0 0x7fbe8413e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:43:00.705 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.699+0000 7fbe8a549700 1 -- 172.21.15.27:0/3511508919 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbe840b3fd0 msgr2=0x7fbe8413e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:43:00.705 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.699+0000 7fbe8a549700 1 --2- 172.21.15.27:0/3511508919 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbe840b3fd0 0x7fbe8413e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:43:00.705 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.699+0000 7fbe8a549700 1 -- 172.21.15.27:0/3511508919 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbe84159500 msgr2=0x7fbe8416d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:43:00.705 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.699+0000 7fbe8a549700 1 --2- 172.21.15.27:0/3511508919 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbe84159500 0x7fbe8416d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:43:00.705 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.699+0000 7fbe8a549700 1 -- 172.21.15.27:0/3511508919 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe7c009ae0 con 0x7fbe8415d8e0 2023-10-06T01:43:00.705 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.699+0000 7fbe8a549700 1 --2- 172.21.15.27:0/3511508919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe8415d8e0 0x7fbe8416db50 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fbe84088d00 tx=0x7fbe7c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:43:00.705 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.699+0000 7fbe81ffb700 1 -- 172.21.15.27:0/3511508919 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe7c0032f0 con 0x7fbe8415d8e0 2023-10-06T01:43:00.705 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.699+0000 7fbe81ffb700 1 -- 172.21.15.27:0/3511508919 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe7c007e30 con 0x7fbe8415d8e0 2023-10-06T01:43:00.705 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.699+0000 7fbe81ffb700 1 -- 172.21.15.27:0/3511508919 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe7c01d670 con 0x7fbe8415d8e0 2023-10-06T01:43:00.706 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.699+0000 7fbe8bfac700 1 -- 172.21.15.27:0/3511508919 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe840a7940 con 0x7fbe8415d8e0 2023-10-06T01:43:00.706 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.700+0000 7fbe8bfac700 1 -- 172.21.15.27:0/3511508919 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe8413e550 con 0x7fbe8415d8e0 2023-10-06T01:43:00.706 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.701+0000 7fbe8bfac700 1 -- 172.21.15.27:0/3511508919 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe840a7e60 con 0x7fbe8415d8e0 2023-10-06T01:43:00.706 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.701+0000 7fbe81ffb700 1 -- 172.21.15.27:0/3511508919 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7fbe7c003040 con 0x7fbe8415d8e0 2023-10-06T01:43:00.706 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.701+0000 7fbe81ffb700 1 --2- 172.21.15.27:0/3511508919 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fbe70066f30 0x7fbe700693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:43:00.706 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.701+0000 7fbe81ffb700 1 -- 172.21.15.27:0/3511508919 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5292+0+0 (secure 0 0 0) 0x7fbe7c085f20 con 0x7fbe8415d8e0 2023-10-06T01:43:00.706 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.702+0000 7fbe89d48700 1 --2- 172.21.15.27:0/3511508919 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fbe70066f30 0x7fbe700693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:43:00.706 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.702+0000 7fbe89d48700 1 --2- 172.21.15.27:0/3511508919 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fbe70066f30 0x7fbe700693e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fbe74006fd0 tx=0x7fbe74008040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:43:00.707 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.706+0000 7fbe81ffb700 1 -- 172.21.15.27:0/3511508919 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbe840a7e60 con 0x7fbe8415d8e0 2023-10-06T01:43:00.900 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.898+0000 7fbe8bfac700 1 -- 172.21.15.27:0/3511508919 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fbe840a7e60 con 0x7fbe8415d8e0 2023-10-06T01:43:00.905 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.902+0000 7fbe81ffb700 1 -- 172.21.15.27:0/3511508919 <== mon.0 v2:172.21.15.27:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fbe7c053330 con 0x7fbe8415d8e0 2023-10-06T01:43:00.905 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.903+0000 7fbe81ffb700 1 -- 172.21.15.27:0/3511508919 <== mon.0 v2:172.21.15.27:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v15)=0 v15) v1 ==== 123+0+0 (secure 0 0 0) 0x7fbe840a7e60 con 0x7fbe8415d8e0 2023-10-06T01:43:00.906 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.905+0000 7fbe8bfac700 1 -- 172.21.15.27:0/3511508919 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fbe70066f30 msgr2=0x7fbe700693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:43:00.906 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.905+0000 7fbe8bfac700 1 --2- 172.21.15.27:0/3511508919 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fbe70066f30 0x7fbe700693e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fbe74006fd0 tx=0x7fbe74008040 comp rx=0 tx=0).stop 2023-10-06T01:43:00.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.905+0000 7fbe8bfac700 1 -- 172.21.15.27:0/3511508919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe8415d8e0 msgr2=0x7fbe8416db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:43:00.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.905+0000 7fbe8bfac700 1 --2- 172.21.15.27:0/3511508919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe8415d8e0 0x7fbe8416db50 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fbe84088d00 tx=0x7fbe7c018000 comp rx=0 tx=0).stop 2023-10-06T01:43:00.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.905+0000 7fbe8bfac700 1 -- 172.21.15.27:0/3511508919 shutdown_connections 2023-10-06T01:43:00.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.905+0000 7fbe8bfac700 1 --2- 172.21.15.27:0/3511508919 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7fbe70066f30 0x7fbe700693e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:43:00.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.905+0000 7fbe8bfac700 1 --2- 172.21.15.27:0/3511508919 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7fbe840b3fd0 0x7fbe8413e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:43:00.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.905+0000 7fbe8bfac700 1 --2- 172.21.15.27:0/3511508919 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7fbe8415d8e0 0x7fbe8416db50 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:43:00.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.905+0000 7fbe8bfac700 1 --2- 172.21.15.27:0/3511508919 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7fbe84159500 0x7fbe8416d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:43:00.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.905+0000 7fbe8bfac700 1 -- 172.21.15.27:0/3511508919 >> 172.21.15.27:0/3511508919 conn(0x7fbe84067630 msgr2=0x7fbe840b5e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:43:00.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.905+0000 7fbe8bfac700 1 -- 172.21.15.27:0/3511508919 shutdown_connections 2023-10-06T01:43:00.907 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:00.905+0000 7fbe8bfac700 1 -- 172.21.15.27:0/3511508919 wait complete. 2023-10-06T01:43:00.918 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-06T01:43:00.918 DEBUG:teuthology.orchestra.run.smithi027:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-06T01:43:01.037 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.036+0000 7f69eb03a700 1 Processor -- start 2023-10-06T01:43:01.037 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.036+0000 7f69eb03a700 1 -- start start 2023-10-06T01:43:01.039 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.036+0000 7f69eb03a700 1 --2- >> v2:172.21.15.124:3300/0 conn(0x7f69e40c8af0 0x7f69e4068640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:43:01.040 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.036+0000 7f69eb03a700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f69e4068b80 0x7f69e4156920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:43:01.040 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.036+0000 7f69eb03a700 1 -- --> v1:172.21.15.124:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69e40acdd0 con 0x7f69e40c8720 2023-10-06T01:43:01.040 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.036+0000 7f69eb03a700 1 -- --> v2:172.21.15.27:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69e4069020 con 0x7f69e4068b80 2023-10-06T01:43:01.040 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.037+0000 7f69eb03a700 1 -- --> v2:172.21.15.124:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69e40ad390 con 0x7f69e40c8af0 2023-10-06T01:43:01.040 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.037+0000 7f69e95d7700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f69e4068b80 0x7f69e4156920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:43:01.040 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.037+0000 7f69e95d7700 1 --2- >> v2:172.21.15.27:3300/0 conn(0x7f69e4068b80 0x7f69e4156920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38702/0 (socket says 172.21.15.27:38702) 2023-10-06T01:43:01.040 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.037+0000 7f69e95d7700 1 -- 172.21.15.27:0/3719127794 learned_addr learned my addr 172.21.15.27:0/3719127794 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:43:01.040 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.037+0000 7f69e3fff700 1 --2- 172.21.15.27:0/3719127794 >> v2:172.21.15.124:3300/0 conn(0x7f69e40c8af0 0x7f69e4068640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:43:01.040 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.037+0000 7f69e95d7700 1 -- 172.21.15.27:0/3719127794 >> v2:172.21.15.124:3300/0 conn(0x7f69e40c8af0 msgr2=0x7f69e4068640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:43:01.040 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.037+0000 7f69e95d7700 1 --2- 172.21.15.27:0/3719127794 >> v2:172.21.15.124:3300/0 conn(0x7f69e40c8af0 0x7f69e4068640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:43:01.040 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.037+0000 7f69e95d7700 1 -- 172.21.15.27:0/3719127794 >> v1:172.21.15.124:6789/0 conn(0x7f69e40c8720 legacy=0x7f69e40ad820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-06T01:43:01.041 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.037+0000 7f69e95d7700 1 -- 172.21.15.27:0/3719127794 --> v2:172.21.15.27:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69e4157560 con 0x7f69e4068b80 2023-10-06T01:43:01.041 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.037+0000 7f69e3fff700 1 --2- 172.21.15.27:0/3719127794 >> v2:172.21.15.124:3300/0 conn(0x7f69e40c8af0 0x7f69e4068640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:43:01.041 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.037+0000 7f69e95d7700 1 --2- 172.21.15.27:0/3719127794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69e4068b80 0x7f69e4156920 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f69e40634c0 tx=0x7f69cc00b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d5feea154d96808 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:43:01.041 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.037+0000 7f69e37fe700 1 -- 172.21.15.27:0/3719127794 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69cc00dc10 con 0x7f69e4068b80 2023-10-06T01:43:01.041 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.037+0000 7f69e37fe700 1 -- 172.21.15.27:0/3719127794 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f69cc00dd70 con 0x7f69e4068b80 2023-10-06T01:43:01.041 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.037+0000 7f69e37fe700 1 -- 172.21.15.27:0/3719127794 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69cc0106e0 con 0x7f69e4068b80 2023-10-06T01:43:01.041 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.038+0000 7f69eb03a700 1 -- 172.21.15.27:0/3719127794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69e4068b80 msgr2=0x7f69e4156920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:43:01.041 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.038+0000 7f69eb03a700 1 --2- 172.21.15.27:0/3719127794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69e4068b80 0x7f69e4156920 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f69e40634c0 tx=0x7f69cc00b4f0 comp rx=0 tx=0).stop 2023-10-06T01:43:01.041 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.038+0000 7f69eb03a700 1 -- 172.21.15.27:0/3719127794 shutdown_connections 2023-10-06T01:43:01.041 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.038+0000 7f69eb03a700 1 --2- 172.21.15.27:0/3719127794 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69e4068b80 0x7f69e4156920 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:43:01.041 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.038+0000 7f69eb03a700 1 --2- 172.21.15.27:0/3719127794 >> v2:172.21.15.124:3300/0 conn(0x7f69e40c8af0 0x7f69e4068640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:43:01.041 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.038+0000 7f69eb03a700 1 -- 172.21.15.27:0/3719127794 >> 172.21.15.27:0/3719127794 conn(0x7f69e40ccd30 msgr2=0x7f69e40694b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:43:01.042 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.038+0000 7f69eb03a700 1 -- 172.21.15.27:0/3719127794 shutdown_connections 2023-10-06T01:43:01.042 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.038+0000 7f69eb03a700 1 -- 172.21.15.27:0/3719127794 wait complete. 2023-10-06T01:43:01.044 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.039+0000 7f69eb03a700 1 Processor -- start 2023-10-06T01:43:01.044 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.039+0000 7f69eb03a700 1 -- start start 2023-10-06T01:43:01.044 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.040+0000 7f69eb03a700 1 --2- >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f69e4068b80 0x7f69e4147c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:43:01.044 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.040+0000 7f69eb03a700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69e40c8720 0x7f69e41541f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:43:01.044 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.040+0000 7f69eb03a700 1 --2- >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f69e40c8af0 0x7f69e40ca150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:43:01.044 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.040+0000 7f69eb03a700 1 -- --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e4155830 con 0x7f69e40c8720 2023-10-06T01:43:01.045 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.040+0000 7f69eb03a700 1 -- --> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69e415a5c0 con 0x7f69e4068b80 2023-10-06T01:43:01.045 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.040+0000 7f69eb03a700 1 -- --> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69e4154cb0 con 0x7f69e40c8af0 2023-10-06T01:43:01.045 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.040+0000 7f69e3fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69e40c8720 0x7f69e41541f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:43:01.045 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.040+0000 7f69e3fff700 1 --2- >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69e40c8720 0x7f69e41541f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.27:3300/0 says I am v2:172.21.15.27:38714/0 (socket says 172.21.15.27:38714) 2023-10-06T01:43:01.045 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.040+0000 7f69e3fff700 1 -- 172.21.15.27:0/5603433 learned_addr learned my addr 172.21.15.27:0/5603433 (peer_addr_for_me v2:172.21.15.27:0/0) 2023-10-06T01:43:01.045 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.040+0000 7f69e8dd6700 1 --2- 172.21.15.27:0/5603433 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f69e4068b80 0x7f69e4147c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:43:01.045 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.040+0000 7f69e95d7700 1 --2- 172.21.15.27:0/5603433 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f69e40c8af0 0x7f69e40ca150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:43:01.045 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.040+0000 7f69e3fff700 1 -- 172.21.15.27:0/5603433 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f69e40c8af0 msgr2=0x7f69e40ca150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:43:01.045 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.040+0000 7f69e3fff700 1 --2- 172.21.15.27:0/5603433 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f69e40c8af0 0x7f69e40ca150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:43:01.045 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.040+0000 7f69e3fff700 1 -- 172.21.15.27:0/5603433 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f69e4068b80 msgr2=0x7f69e4147c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:43:01.045 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.040+0000 7f69e3fff700 1 --2- 172.21.15.27:0/5603433 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f69e4068b80 0x7f69e4147c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:43:01.045 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.040+0000 7f69e3fff700 1 -- 172.21.15.27:0/5603433 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69cc00b790 con 0x7f69e40c8720 2023-10-06T01:43:01.046 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.040+0000 7f69e95d7700 1 --2- 172.21.15.27:0/5603433 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f69e40c8af0 0x7f69e40ca150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-06T01:43:01.046 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.040+0000 7f69e3fff700 1 --2- 172.21.15.27:0/5603433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69e40c8720 0x7f69e41541f0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f69d400c880 tx=0x7f69d4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:43:01.046 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.041+0000 7f69e0ff9700 1 -- 172.21.15.27:0/5603433 <== mon.0 v2:172.21.15.27:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d4011600 con 0x7f69e40c8720 2023-10-06T01:43:01.046 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.041+0000 7f69e0ff9700 1 -- 172.21.15.27:0/5603433 <== mon.0 v2:172.21.15.27:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f69d4011760 con 0x7f69e40c8720 2023-10-06T01:43:01.046 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.041+0000 7f69eb03a700 1 -- 172.21.15.27:0/5603433 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69e4154f90 con 0x7f69e40c8720 2023-10-06T01:43:01.046 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.041+0000 7f69e0ff9700 1 -- 172.21.15.27:0/5603433 <== mon.0 v2:172.21.15.27:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d4011b00 con 0x7f69e40c8720 2023-10-06T01:43:01.046 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.041+0000 7f69eb03a700 1 -- 172.21.15.27:0/5603433 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69e4155550 con 0x7f69e40c8720 2023-10-06T01:43:01.046 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.042+0000 7f69eb03a700 1 -- 172.21.15.27:0/5603433 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69e4064050 con 0x7f69e40c8720 2023-10-06T01:43:01.046 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.042+0000 7f69e0ff9700 1 -- 172.21.15.27:0/5603433 <== mon.0 v2:172.21.15.27:3300/0 4 ==== mgrmap(e 5) v1 ==== 84740+0+0 (secure 0 0 0) 0x7f69d40073d0 con 0x7f69e40c8720 2023-10-06T01:43:01.046 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.042+0000 7f69e0ff9700 1 --2- 172.21.15.27:0/5603433 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f69c4066f30 0x7f69c40693e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-06T01:43:01.046 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.043+0000 7f69e0ff9700 1 -- 172.21.15.27:0/5603433 <== mon.0 v2:172.21.15.27:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5292+0+0 (secure 0 0 0) 0x7f69d4084400 con 0x7f69e40c8720 2023-10-06T01:43:01.047 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.043+0000 7f69e8dd6700 1 --2- 172.21.15.27:0/5603433 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f69c4066f30 0x7f69c40693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-06T01:43:01.047 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.043+0000 7f69e8dd6700 1 --2- 172.21.15.27:0/5603433 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f69c4066f30 0x7f69c40693e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f69dc007fd0 tx=0x7f69dc009040 comp rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-06T01:43:01.048 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.047+0000 7f69e0ff9700 1 -- 172.21.15.27:0/5603433 <== mon.0 v2:172.21.15.27:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f69d4051890 con 0x7f69e40c8720 2023-10-06T01:43:01.286 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.285+0000 7f69eb03a700 1 -- 172.21.15.27:0/5603433 --> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69e40cbc80 con 0x7f69e40c8720 2023-10-06T01:43:01.287 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.285+0000 7f69e0ff9700 1 -- 172.21.15.27:0/5603433 <== mon.0 v2:172.21.15.27:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+3532 (secure 0 0 0) 0x7f69d4015de0 con 0x7f69e40c8720 2023-10-06T01:43:01.287 INFO:teuthology.orchestra.run.smithi027.stdout: 2023-10-06T01:43:01.287 INFO:teuthology.orchestra.run.smithi027.stdout:{"epoch":18,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4870,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6836/1043044736","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6834","nonce":1043044736},{"type":"v1","addr":"172.21.15.124:6836","nonce":1043044736}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4962,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6834/1207987117","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6832","nonce":1207987117},{"type":"v1","addr":"172.21.15.27:6834","nonce":1207987117}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":4965,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.27:6835/943261840","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.27:6833","nonce":943261840},{"type":"v1","addr":"172.21.15.27:6835","nonce":943261840}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":5281,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.124:6837/1275107386","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.124:6835","nonce":1275107386},{"type":"v1","addr":"172.21.15.124:6837","nonce":1275107386}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[]} 2023-10-06T01:43:01.289 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.287+0000 7f69eb03a700 1 -- 172.21.15.27:0/5603433 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f69c4066f30 msgr2=0x7f69c40693e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:43:01.289 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.287+0000 7f69eb03a700 1 --2- 172.21.15.27:0/5603433 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f69c4066f30 0x7f69c40693e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f69dc007fd0 tx=0x7f69dc009040 comp rx=0 tx=0).stop 2023-10-06T01:43:01.289 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.287+0000 7f69eb03a700 1 -- 172.21.15.27:0/5603433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69e40c8720 msgr2=0x7f69e41541f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-06T01:43:01.289 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.287+0000 7f69eb03a700 1 --2- 172.21.15.27:0/5603433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69e40c8720 0x7f69e41541f0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f69d400c880 tx=0x7f69d4008040 comp rx=0 tx=0).stop 2023-10-06T01:43:01.289 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.287+0000 7f69eb03a700 1 -- 172.21.15.27:0/5603433 shutdown_connections 2023-10-06T01:43:01.289 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.287+0000 7f69eb03a700 1 --2- 172.21.15.27:0/5603433 >> [v2:172.21.15.124:6832/90278,v1:172.21.15.124:6833/90278] conn(0x7f69c4066f30 0x7f69c40693e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:43:01.289 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.287+0000 7f69eb03a700 1 --2- 172.21.15.27:0/5603433 >> [v2:172.21.15.124:3301/0,v1:172.21.15.124:6790/0] conn(0x7f69e40c8af0 0x7f69e40ca150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:43:01.290 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.287+0000 7f69eb03a700 1 --2- 172.21.15.27:0/5603433 >> [v2:172.21.15.124:3300/0,v1:172.21.15.124:6789/0] conn(0x7f69e4068b80 0x7f69e4147c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:43:01.290 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.287+0000 7f69eb03a700 1 --2- 172.21.15.27:0/5603433 >> [v2:172.21.15.27:3300/0,v1:172.21.15.27:6789/0] conn(0x7f69e40c8720 0x7f69e41541f0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-06T01:43:01.290 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.287+0000 7f69eb03a700 1 -- 172.21.15.27:0/5603433 >> 172.21.15.27:0/5603433 conn(0x7f69e40ccd30 msgr2=0x7f69e40c91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-06T01:43:01.290 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.287+0000 7f69eb03a700 1 -- 172.21.15.27:0/5603433 shutdown_connections 2023-10-06T01:43:01.290 INFO:teuthology.orchestra.run.smithi027.stderr:2023-10-06T01:43:01.288+0000 7f69eb03a700 1 -- 172.21.15.27:0/5603433 wait complete. 2023-10-06T01:43:01.290 INFO:teuthology.orchestra.run.smithi027.stderr:dumped fsmap epoch 18 2023-10-06T01:43:01.301 DEBUG:tasks.cephfs.filesystem:already dead... 2023-10-06T01:43:01.301 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-06T01:43:01.301 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-06T01:43:01.301 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T01:43:01.310 INFO:tasks.ceph.mds.a:Stopped 2023-10-06T01:43:01.310 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-06T01:43:01.310 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T01:43:01.359 INFO:tasks.ceph.mds.c:Stopped 2023-10-06T01:43:01.359 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-06T01:43:01.359 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T01:43:01.369 INFO:tasks.ceph.mds.b:Stopped 2023-10-06T01:43:01.370 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-06T01:43:01.370 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T01:43:01.421 INFO:tasks.ceph.mds.d:Stopped 2023-10-06T01:43:01.421 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-06T01:43:01.421 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-06T01:43:01.421 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T01:43:01.446 INFO:tasks.ceph.osd.0:Stopped 2023-10-06T01:43:01.446 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-06T01:43:01.447 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T01:43:01.513 INFO:tasks.ceph.osd.1:Stopped 2023-10-06T01:43:01.513 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-06T01:43:01.513 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T01:43:01.581 INFO:tasks.ceph.osd.2:Stopped 2023-10-06T01:43:01.581 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-10-06T01:43:01.581 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T01:43:01.644 INFO:tasks.ceph.osd.3:Stopped 2023-10-06T01:43:01.645 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-10-06T01:43:01.645 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T01:43:01.694 INFO:tasks.ceph.osd.4:Stopped 2023-10-06T01:43:01.694 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-10-06T01:43:01.694 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T01:43:01.738 INFO:tasks.ceph.osd.5:Stopped 2023-10-06T01:43:01.738 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-10-06T01:43:01.738 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T01:43:01.780 INFO:tasks.ceph.osd.6:Stopped 2023-10-06T01:43:01.780 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-10-06T01:43:01.780 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T01:43:01.824 INFO:tasks.ceph.osd.7:Stopped 2023-10-06T01:43:01.824 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-06T01:43:01.824 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-10-06T01:43:01.824 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T01:43:01.846 INFO:tasks.ceph.mgr.y:Stopped 2023-10-06T01:43:01.847 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-06T01:43:01.847 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T01:43:01.873 INFO:tasks.ceph.mgr.x:Stopped 2023-10-06T01:43:01.873 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-06T01:43:01.873 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-06T01:43:01.874 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T01:43:01.897 INFO:tasks.ceph.mon.a:Stopped 2023-10-06T01:43:01.898 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-10-06T01:43:01.898 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T01:43:01.924 INFO:tasks.ceph.mon.b:Stopped 2023-10-06T01:43:01.924 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-10-06T01:43:01.924 INFO:teuthology.orchestra.run:waiting for 300 2023-10-06T01:43:01.975 INFO:tasks.ceph.mon.c:Stopped 2023-10-06T01:43:01.975 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-06T01:43:01.976 DEBUG:teuthology.orchestra.run.smithi027:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-10-06T01:43:02.005 INFO:teuthology.orchestra.run.smithi027.stdout:1696556472.3133135 mon.a (mon.0) 135 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-10-06T01:43:02.005 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-10-06T01:43:02.005 DEBUG:teuthology.orchestra.run.smithi027:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-10-06T01:43:02.073 DEBUG:teuthology.orchestra.run.smithi027:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-10-06T01:43:02.142 DEBUG:teuthology.orchestra.run.smithi027:> sudo egrep '\[WRN\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-10-06T01:43:02.211 INFO:teuthology.orchestra.run.smithi027.stdout:1696556472.3133135 mon.a (mon.0) 135 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-10-06T01:43:02.211 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi027.front.sepia.ceph.com 2023-10-06T01:43:02.212 DEBUG:teuthology.orchestra.run.smithi027:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-06T01:43:02.429 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi027.front.sepia.ceph.com 2023-10-06T01:43:02.430 DEBUG:teuthology.orchestra.run.smithi027:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-06T01:43:02.519 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi027.front.sepia.ceph.com 2023-10-06T01:43:02.520 DEBUG:teuthology.orchestra.run.smithi027:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-06T01:43:02.582 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi027.front.sepia.ceph.com 2023-10-06T01:43:02.583 DEBUG:teuthology.orchestra.run.smithi027:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-10-06T01:43:02.642 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi124.front.sepia.ceph.com 2023-10-06T01:43:02.642 DEBUG:teuthology.orchestra.run.smithi124:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-10-06T01:43:02.813 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi124.front.sepia.ceph.com 2023-10-06T01:43:02.814 DEBUG:teuthology.orchestra.run.smithi124:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-10-06T01:43:02.888 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi124.front.sepia.ceph.com 2023-10-06T01:43:02.888 DEBUG:teuthology.orchestra.run.smithi124:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-10-06T01:43:02.947 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi124.front.sepia.ceph.com 2023-10-06T01:43:02.948 DEBUG:teuthology.orchestra.run.smithi124:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-10-06T01:43:03.015 INFO:tasks.ceph:Archiving mon data... 2023-10-06T01:43:03.016 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi/7412096/data/mon.a.tgz 2023-10-06T01:43:03.016 DEBUG:teuthology.orchestra.run.smithi027:> mktemp 2023-10-06T01:43:03.032 INFO:teuthology.orchestra.run.smithi027.stdout:/tmp/tmp.kVRra2JG1b 2023-10-06T01:43:03.032 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.kVRra2JG1b 2023-10-06T01:43:03.169 DEBUG:teuthology.orchestra.run.smithi027:> sudo chmod 0666 /tmp/tmp.kVRra2JG1b 2023-10-06T01:43:03.251 DEBUG:teuthology.orchestra.remote:smithi027:/tmp/tmp.kVRra2JG1b is 278KB 2023-10-06T01:43:03.345 DEBUG:teuthology.orchestra.run.smithi027:> rm -fr /tmp/tmp.kVRra2JG1b 2023-10-06T01:43:03.360 DEBUG:teuthology.misc:Transferring archived files from smithi124:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi/7412096/data/mon.b.tgz 2023-10-06T01:43:03.360 DEBUG:teuthology.orchestra.run.smithi124:> mktemp 2023-10-06T01:43:03.375 INFO:teuthology.orchestra.run.smithi124.stdout:/tmp/tmp.NlnwmSB8FX 2023-10-06T01:43:03.375 DEBUG:teuthology.orchestra.run.smithi124:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.NlnwmSB8FX 2023-10-06T01:43:03.510 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod 0666 /tmp/tmp.NlnwmSB8FX 2023-10-06T01:43:03.591 DEBUG:teuthology.orchestra.remote:smithi124:/tmp/tmp.NlnwmSB8FX is 370KB 2023-10-06T01:43:03.656 DEBUG:teuthology.orchestra.run.smithi124:> rm -fr /tmp/tmp.NlnwmSB8FX 2023-10-06T01:43:03.670 DEBUG:teuthology.misc:Transferring archived files from smithi124:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi/7412096/data/mon.c.tgz 2023-10-06T01:43:03.670 DEBUG:teuthology.orchestra.run.smithi124:> mktemp 2023-10-06T01:43:03.725 INFO:teuthology.orchestra.run.smithi124.stdout:/tmp/tmp.wpVucWYXs7 2023-10-06T01:43:03.725 DEBUG:teuthology.orchestra.run.smithi124:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.wpVucWYXs7 2023-10-06T01:43:03.854 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod 0666 /tmp/tmp.wpVucWYXs7 2023-10-06T01:43:03.935 DEBUG:teuthology.orchestra.remote:smithi124:/tmp/tmp.wpVucWYXs7 is 370KB 2023-10-06T01:43:04.030 DEBUG:teuthology.orchestra.run.smithi124:> rm -fr /tmp/tmp.wpVucWYXs7 2023-10-06T01:43:04.045 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-06T01:43:04.045 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-06T01:43:04.047 DEBUG:teuthology.orchestra.run.smithi124:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-06T01:43:04.107 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-06T01:43:04.107 DEBUG:teuthology.orchestra.run.smithi027:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-06T01:43:04.114 DEBUG:teuthology.orchestra.run.smithi124:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-06T01:43:04.139 INFO:teuthology.orchestra.run.smithi027.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-06T01:43:04.171 INFO:teuthology.orchestra.run.smithi124.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-06T01:43:04.177 INFO:tasks.ceph:Archiving crash dumps... 2023-10-06T01:43:04.178 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi/7412096/remote/smithi027/crash 2023-10-06T01:43:04.179 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-06T01:43:04.212 DEBUG:teuthology.misc:Transferring archived files from smithi124:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi/7412096/remote/smithi124/crash 2023-10-06T01:43:04.213 DEBUG:teuthology.orchestra.run.smithi124:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-06T01:43:04.240 INFO:tasks.ceph:Shutting down logrotate 2023-10-06T01:43:04.240 DEBUG:teuthology.orchestra.run.smithi027:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T01:43:04.256 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-06T01:43:04.336 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-06T01:43:04.361 DEBUG:teuthology.orchestra.run.smithi124:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-06T01:43:04.384 INFO:tasks.ceph:Compressing logs... 2023-10-06T01:43:04.385 DEBUG:teuthology.orchestra.run.smithi027:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-06T01:43:04.403 DEBUG:teuthology.orchestra.run.smithi124:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-06T01:43:05.282 INFO:tasks.ceph:Archiving logs... 2023-10-06T01:43:05.283 DEBUG:teuthology.misc:Transferring archived files from smithi027:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi/7412096/remote/smithi027/log 2023-10-06T01:43:05.284 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-06T01:43:05.559 DEBUG:teuthology.misc:Transferring archived files from smithi124:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi/7412096/remote/smithi124/log 2023-10-06T01:43:05.560 DEBUG:teuthology.orchestra.run.smithi124:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-06T01:43:05.695 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-06T01:43:05.709 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-06T01:43:05.709 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-06T01:43:05.711 DEBUG:teuthology.orchestra.run.smithi124:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-06T01:43:05.795 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-06T01:43:05.804 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-06T01:43:05.804 DEBUG:teuthology.orchestra.run.smithi027:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-06T01:43:05.806 DEBUG:teuthology.orchestra.run.smithi124:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-06T01:43:05.818 INFO:teuthology.orchestra.run.smithi027.stderr:bash: ntpq: command not found 2023-10-06T01:43:05.819 INFO:teuthology.orchestra.run.smithi124.stderr:bash: ntpq: command not found 2023-10-06T01:43:05.821 INFO:teuthology.orchestra.run.smithi027.stdout:210 Number of sources = 4 2023-10-06T01:43:05.821 INFO:teuthology.orchestra.run.smithi027.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-06T01:43:05.821 INFO:teuthology.orchestra.run.smithi027.stdout:=============================================================================== 2023-10-06T01:43:05.821 INFO:teuthology.orchestra.run.smithi027.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 29 +42us[ +53us] +/- 77ms 2023-10-06T01:43:05.822 INFO:teuthology.orchestra.run.smithi027.stdout:^* hv02.front.sepia.ceph.com 3 6 377 29 -413us[ -402us] +/- 65ms 2023-10-06T01:43:05.822 INFO:teuthology.orchestra.run.smithi027.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 29 +437us[ +448us] +/- 72ms 2023-10-06T01:43:05.822 INFO:teuthology.orchestra.run.smithi027.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-10-06T01:43:05.823 INFO:teuthology.orchestra.run.smithi124.stdout:210 Number of sources = 4 2023-10-06T01:43:05.823 INFO:teuthology.orchestra.run.smithi124.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-06T01:43:05.824 INFO:teuthology.orchestra.run.smithi124.stdout:=============================================================================== 2023-10-06T01:43:05.824 INFO:teuthology.orchestra.run.smithi124.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 28 +41us[ +41us] +/- 77ms 2023-10-06T01:43:05.824 INFO:teuthology.orchestra.run.smithi124.stdout:^* hv02.front.sepia.ceph.com 3 6 377 30 -411us[ -415us] +/- 65ms 2023-10-06T01:43:05.824 INFO:teuthology.orchestra.run.smithi124.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 29 +431us[ +431us] +/- 72ms 2023-10-06T01:43:05.824 INFO:teuthology.orchestra.run.smithi124.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-10-06T01:43:05.824 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-06T01:43:05.835 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-06T01:43:05.835 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-06T01:43:05.845 DEBUG:teuthology.orchestra.run.smithi027:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-06T01:43:05.961 DEBUG:teuthology.orchestra.run.smithi124:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-06T01:43:06.049 DEBUG:teuthology.orchestra.run.smithi027:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-06T01:43:06.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T01:43:06.080 DEBUG:teuthology.orchestra.run.smithi124:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-06T01:43:06.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T01:43:06.109 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-06T01:43:06.119 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-06T01:43:06.128 INFO:teuthology.task.internal:Duration was 796.980628 seconds 2023-10-06T01:43:06.128 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-06T01:43:06.138 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-06T01:43:06.138 DEBUG:teuthology.orchestra.run.smithi027:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-06T01:43:06.141 DEBUG:teuthology.orchestra.run.smithi124:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-06T01:43:06.184 INFO:teuthology.orchestra.run.smithi027.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-06T01:43:06.193 INFO:teuthology.orchestra.run.smithi124.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-06T01:43:06.450 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-06T01:43:06.450 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi027.front.sepia.ceph.com 2023-10-06T01:43:06.450 DEBUG:teuthology.orchestra.run.smithi027:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-06T01:43:06.470 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi124.front.sepia.ceph.com 2023-10-06T01:43:06.470 DEBUG:teuthology.orchestra.run.smithi124:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-06T01:43:06.488 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-06T01:43:06.488 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-06T01:43:06.512 DEBUG:teuthology.orchestra.run.smithi124:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-06T01:43:06.567 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-06T01:43:06.567 DEBUG:teuthology.orchestra.run.smithi027:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-06T01:43:06.593 DEBUG:teuthology.orchestra.run.smithi124:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-06T01:43:06.680 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-06T01:43:06.690 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-06T01:43:06.690 DEBUG:teuthology.orchestra.run.smithi027:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-06T01:43:06.715 DEBUG:teuthology.orchestra.run.smithi124:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-06T01:43:06.739 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-06T01:43:06.749 DEBUG:teuthology.orchestra.run.smithi027:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-06T01:43:06.757 DEBUG:teuthology.orchestra.run.smithi124:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-06T01:43:06.779 INFO:teuthology.orchestra.run.smithi027.stdout:kernel.core_pattern = core 2023-10-06T01:43:06.803 INFO:teuthology.orchestra.run.smithi124.stdout:kernel.core_pattern = core 2023-10-06T01:43:06.817 DEBUG:teuthology.orchestra.run.smithi027:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-06T01:43:06.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T01:43:06.848 DEBUG:teuthology.orchestra.run.smithi124:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-06T01:43:06.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-06T01:43:06.871 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-06T01:43:06.881 INFO:teuthology.task.internal:Transferring archived files... 2023-10-06T01:43:06.882 DEBUG:teuthology.misc:Transferring archived files from smithi027:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi/7412096/remote/smithi027 2023-10-06T01:43:06.882 DEBUG:teuthology.orchestra.run.smithi027:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-06T01:43:06.924 DEBUG:teuthology.misc:Transferring archived files from smithi124:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi/7412096/remote/smithi124 2023-10-06T01:43:06.924 DEBUG:teuthology.orchestra.run.smithi124:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-06T01:43:06.958 INFO:teuthology.task.internal:Removing archive directory... 2023-10-06T01:43:06.958 DEBUG:teuthology.orchestra.run.smithi027:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-06T01:43:06.960 DEBUG:teuthology.orchestra.run.smithi124:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-06T01:43:07.007 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-06T01:43:07.019 INFO:teuthology.task.internal:Not uploading archives. 2023-10-06T01:43:07.019 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-06T01:43:07.029 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-06T01:43:07.029 DEBUG:teuthology.orchestra.run.smithi027:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-06T01:43:07.031 DEBUG:teuthology.orchestra.run.smithi124:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-06T01:43:07.044 INFO:teuthology.orchestra.run.smithi027.stdout: 262176 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 6 01:43 /home/ubuntu/cephtest 2023-10-06T01:43:07.062 INFO:teuthology.orchestra.run.smithi124.stdout: 262153 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 6 01:43 /home/ubuntu/cephtest 2023-10-06T01:43:07.063 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-06T01:43:07.074 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-06T01:43:07.096 INFO:teuthology.nuke:Checking targets against current locks 2023-10-06T01:43:07.121 DEBUG:teuthology.nuke:shortname: smithi027 2023-10-06T01:43:07.121 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-06T01:43:07.147 DEBUG:teuthology.nuke:shortname: smithi124 2023-10-06T01:43:07.147 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-06T01:43:07.151 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi027.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi/7412096', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-06 01:20:35.180778', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKMVItaKZieNJGHWDSdhW5L1jqh1+xgxATt847HawcIgu7996RoKJqcfPy6oEBctp/7TfmFHn08ST1pb4RT6P/M='} 2023-10-06T01:43:07.173 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi124.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-10-05_23:52:50-fs-wip-yuri-testing-2023-10-05-0730-quincy-distro-default-smithi/7412096', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-06 01:20:35.179337', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPlciKcQZmKpVk5mhkr/65Bra2WtrkfGj0rG/nL0nqbOlucdiZdD+gY5t7TaSup1xruJNSuNkeYmPg9GEdBjrmo='} 2023-10-06T01:43:07.203 INFO:teuthology.orchestra.console.smithi027:Power off 2023-10-06T01:43:07.203 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-06T01:43:07.224 DEBUG:teuthology.orchestra.console.smithi027:power off output: Chassis Power Control: Down/Off 2023-10-06T01:43:07.225 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-06T01:43:07.239 INFO:teuthology.orchestra.console.smithi124:Power off 2023-10-06T01:43:07.239 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-06T01:43:07.247 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is on 2023-10-06T01:43:07.262 DEBUG:teuthology.orchestra.console.smithi124:power off output: Chassis Power Control: Down/Off 2023-10-06T01:43:07.262 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-06T01:43:07.286 DEBUG:teuthology.orchestra.console.smithi124:check power output: Chassis Power is on 2023-10-06T01:43:11.248 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-06T01:43:11.288 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-06T01:43:11.357 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is on 2023-10-06T01:43:11.397 DEBUG:teuthology.orchestra.console.smithi124:check power output: Chassis Power is on 2023-10-06T01:43:15.361 DEBUG:teuthology.orchestra.console.smithi027:pexpect command: ipmitool -H smithi027.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-06T01:43:15.399 DEBUG:teuthology.orchestra.console.smithi124:pexpect command: ipmitool -H smithi124.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-06T01:43:15.469 DEBUG:teuthology.orchestra.console.smithi027:check power output: Chassis Power is off 2023-10-06T01:43:15.507 DEBUG:teuthology.orchestra.console.smithi124:check power output: Chassis Power is off 2023-10-06T01:43:15.571 INFO:teuthology.orchestra.console.smithi027:Power off completed 2023-10-06T01:43:15.608 INFO:teuthology.orchestra.console.smithi124:Power off completed 2023-10-06T01:43:15.713 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{rhel_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/backtrace} duration: 796.9806282520294 failure_reason: '"1696556472.3133135 mon.a (mon.0) 135 : cluster [WRN] Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED)" in cluster log' flavor: default owner: scheduled_yuriw@teuthology success: false 2023-10-06T01:43:15.713 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-06T01:43:15.782 INFO:teuthology.run:FAIL